Rifqi Hafizuddin Claude Fable 5 commited on
Commit ·
87cfcf8
1
Parent(s): ef0c7b1
[KM-715] charts: render_chart tool + message_charts store + GET /api/v1/charts + planner viz slice
Browse filesW2 of SPINE_V2_PLAN: deterministic Plotly-JSON spec builder (dataeyond.chart.v1,
no plotly dep), ToolOutput.kind "chart", never-throw ChartStore mirroring
traceability (rows written before done), FE fetches GET /charts on done (empty
list = valid 200; done event unchanged). Planner: recipe table + explicit-ask
rule, Examples J (viz tail) + K (viz-infeasible), validator Check 10
(render_chart.data must be table-kind). Traceability records charts compactly
(point_count, not raw arrays). Live e2e vs dedorch: ALL PASS.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- API_CONTRACT_BE_PYTHON.md +91 -0
- main.py +2 -0
- src/agents/chat_handler.py +28 -0
- src/agents/planner/examples.py +115 -0
- src/agents/planner/validator.py +18 -0
- src/api/v1/charts.py +46 -0
- src/charts/__init__.py +21 -0
- src/charts/store.py +154 -0
- src/config/prompts/planner.md +32 -1
- src/db/postgres/models.py +33 -0
- src/tools/analytics/visualization.py +239 -0
- src/tools/contracts.py +1 -1
- src/tools/invoker.py +2 -0
- src/tools/registry.py +20 -1
- src/traceability/scratchpad.py +33 -1
API_CONTRACT_BE_PYTHON.md
CHANGED
|
@@ -34,6 +34,7 @@ The frontend uses this service during the analysis conversation flow:
|
|
| 34 |
| `GET` | `/api/v1/tools/report/{analysis_id}/readiness` | Report-readiness signal for the Generate-Report button (added 2026-07-09). |
|
| 35 |
| `GET` | `/api/v1/tools/report/{analysis_id}/{version}` | Retrieve one report version. |
|
| 36 |
| `GET` | `/api/v1/traceability` | Retrieve provenance for one assistant answer. |
|
|
|
|
| 37 |
|
| 38 |
## Common Concepts
|
| 39 |
|
|
@@ -61,6 +62,8 @@ Common event types:
|
|
| 61 |
|
| 62 |
The stream carries answer text only. Planning, tool call details, and full provenance are fetched from `GET /api/v1/traceability` after the stream is done.
|
| 63 |
|
|
|
|
|
|
|
| 64 |
## Chat
|
| 65 |
|
| 66 |
### `POST /api/v2/chat/stream`
|
|
@@ -643,3 +646,91 @@ Frontend rendering guidance:
|
|
| 643 |
- Default state can be collapsed.
|
| 644 |
- Show planning, tool calls, and sources as separate sections.
|
| 645 |
- Treat `planning: null`, `tool_calls: []`, and `sources: []` as valid states.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
| `GET` | `/api/v1/tools/report/{analysis_id}/readiness` | Report-readiness signal for the Generate-Report button (added 2026-07-09). |
|
| 35 |
| `GET` | `/api/v1/tools/report/{analysis_id}/{version}` | Retrieve one report version. |
|
| 36 |
| `GET` | `/api/v1/traceability` | Retrieve provenance for one assistant answer. |
|
| 37 |
+
| `GET` | `/api/v1/charts` | Retrieve chart(s) produced by `render_chart` for one assistant answer (added 2026-07-13). |
|
| 38 |
|
| 39 |
## Common Concepts
|
| 40 |
|
|
|
|
| 62 |
|
| 63 |
The stream carries answer text only. Planning, tool call details, and full provenance are fetched from `GET /api/v1/traceability` after the stream is done.
|
| 64 |
|
| 65 |
+
**Charts (added 2026-07-13, SPINE_V2_PLAN §4.5):** the `done` event is unchanged by charts — no additive `chart_count`/`chart_ids` field yet (open question owned by Harry). Until that's resolved, the frontend fetches `GET /api/v1/charts` unconditionally on every `done`, the same fetch-on-`done` pattern already used for traceability. SSE order and every existing field stay exactly as documented above; `sources` stays `[]`.
|
| 66 |
+
|
| 67 |
## Chat
|
| 68 |
|
| 69 |
### `POST /api/v2/chat/stream`
|
|
|
|
| 646 |
- Default state can be collapsed.
|
| 647 |
- Show planning, tool calls, and sources as separate sections.
|
| 648 |
- Treat `planning: null`, `tool_calls: []`, and `sources: []` as valid states.
|
| 649 |
+
|
| 650 |
+
## Charts
|
| 651 |
+
|
| 652 |
+
> Added 2026-07-13 (SPINE_V2_PLAN §4.4/§4.5, S2 visualization). A chart is a `render_chart` tool output, planner-selected only when the user explicitly asks to plot/visualize. Delivery mirrors traceability: a Python-owned store plus a dedicated `GET` endpoint; the streamed answer (SSE) stays text-only — charts are never embedded in `chunk`.
|
| 653 |
+
|
| 654 |
+
### `GET /api/v1/charts`
|
| 655 |
+
|
| 656 |
+
Returns every chart produced by `render_chart` during one assistant answer.
|
| 657 |
+
|
| 658 |
+
The frontend should call this after the chat stream emits `done`, using the `analysis_id` + `message_id` from the `done` event — the same fetch-on-`done` pattern as traceability; the row(s) are written **before** `done`, so there is no polling race. Unlike `GET /api/v1/traceability`, **an empty list is a valid `200`** — most turns don't ask for a chart, and once `done` has fired there is no "not ready yet" state to distinguish from "no charts this turn", so this endpoint never 404s on a chartless turn.
|
| 659 |
+
|
| 660 |
+
Query params:
|
| 661 |
+
|
| 662 |
+
| Query | Required | Description |
|
| 663 |
+
| --- | --- | --- |
|
| 664 |
+
| `analysis_id` | Yes | Analysis identifier. |
|
| 665 |
+
| `message_id` | Yes | Assistant answer identifier returned by the stream. |
|
| 666 |
+
|
| 667 |
+
Example:
|
| 668 |
+
|
| 669 |
+
```text
|
| 670 |
+
GET /api/v1/charts?analysis_id=an_42&message_id=msg_88f1
|
| 671 |
+
```
|
| 672 |
+
|
| 673 |
+
The `dataeyond.chart.v1` envelope (the shape of `charts[].spec`, verbatim from `render_chart`):
|
| 674 |
+
|
| 675 |
+
```json
|
| 676 |
+
{
|
| 677 |
+
"schema": "dataeyond.chart.v1",
|
| 678 |
+
"chart_type": "bar",
|
| 679 |
+
"title": "Revenue by region",
|
| 680 |
+
"plotly": {
|
| 681 |
+
"data": [{ "type": "bar", "x": ["A", "B"], "y": [1, 2], "name": "revenue" }],
|
| 682 |
+
"layout": { "title": {"text": "Revenue by region"}, "xaxis": {"title": {"text": "region"}},
|
| 683 |
+
"yaxis": {"title": {"text": "revenue"}} }
|
| 684 |
+
}
|
| 685 |
+
}
|
| 686 |
+
```
|
| 687 |
+
|
| 688 |
+
Frontend renders it with `Plotly.newPlot(el, spec.plotly.data, spec.plotly.layout)`. v1 chart types: `bar`, `line`, `pie`, `scatter`.
|
| 689 |
+
|
| 690 |
+
Response `200` (one chart):
|
| 691 |
+
|
| 692 |
+
```json
|
| 693 |
+
{
|
| 694 |
+
"count": 1,
|
| 695 |
+
"charts": [
|
| 696 |
+
{
|
| 697 |
+
"chart_id": "3fbd8e2e-8e21-4d4b-9b21-9e6b6a0a6a6e",
|
| 698 |
+
"chart_type": "bar",
|
| 699 |
+
"title": "Revenue by region",
|
| 700 |
+
"spec": {
|
| 701 |
+
"schema": "dataeyond.chart.v1",
|
| 702 |
+
"chart_type": "bar",
|
| 703 |
+
"title": "Revenue by region",
|
| 704 |
+
"plotly": {
|
| 705 |
+
"data": [{ "type": "bar", "x": ["A", "B"], "y": [1, 2], "name": "revenue" }],
|
| 706 |
+
"layout": { "title": {"text": "Revenue by region"}, "xaxis": {"title": {"text": "region"}},
|
| 707 |
+
"yaxis": {"title": {"text": "revenue"}} }
|
| 708 |
+
}
|
| 709 |
+
},
|
| 710 |
+
"created_at": "2026-07-13T03:21:09.114Z"
|
| 711 |
+
}
|
| 712 |
+
]
|
| 713 |
+
}
|
| 714 |
+
```
|
| 715 |
+
|
| 716 |
+
Response `200` (chartless turn — the common case; not an error):
|
| 717 |
+
|
| 718 |
+
```json
|
| 719 |
+
{
|
| 720 |
+
"count": 0,
|
| 721 |
+
"charts": []
|
| 722 |
+
}
|
| 723 |
+
```
|
| 724 |
+
|
| 725 |
+
Field rules:
|
| 726 |
+
|
| 727 |
+
- `spec` is the full `dataeyond.chart.v1` envelope, unmodified — it is the source of truth, not a projection; render straight from it.
|
| 728 |
+
- `chart_type` / `title` are copied out of `spec` for convenience (list rendering without parsing `spec`); `title` may be `null`.
|
| 729 |
+
- A turn can produce more than one chart (multiple `render_chart` calls in the same plan); `charts` is ordered by creation time.
|
| 730 |
+
- The payload carries no `user_id` (unlike traceability) — charts are keyed by `analysis_id` + `message_id` only.
|
| 731 |
+
|
| 732 |
+
Frontend rendering guidance:
|
| 733 |
+
|
| 734 |
+
- Fetch unconditionally on every `done` — see the SSE note above (no `chart_count` hint yet); an empty `charts[]` means render nothing extra.
|
| 735 |
+
- Render each chart under the assistant message it belongs to, via `Plotly.newPlot`.
|
| 736 |
+
- "Chart iteration" v1 = a follow-up chat turn (e.g. "make it a line chart") — the planner re-emits `render_chart` with patched args. There is no separate edit endpoint.
|
main.py
CHANGED
|
@@ -22,6 +22,7 @@ from src.api.v1.report import router as report_router
|
|
| 22 |
from src.api.v1.tools import router as tools_router
|
| 23 |
from src.api.v1.help import router as help_router # pr/5 Phase 2: dedicated /tools/help
|
| 24 |
from src.api.v1.traceability import router as traceability_router # KM-691
|
|
|
|
| 25 |
from src.api.v2.chat import router as chat_v2_router # pr/5 Phase 2: v2 chat pilot (analysis_id)
|
| 26 |
from src.db.postgres.init_db import init_db
|
| 27 |
from src.config.settings import settings
|
|
@@ -68,6 +69,7 @@ app.include_router(report_router)
|
|
| 68 |
app.include_router(tools_router)
|
| 69 |
app.include_router(help_router)
|
| 70 |
app.include_router(traceability_router) # KM-691: GET /api/v1/traceability
|
|
|
|
| 71 |
app.include_router(chat_v2_router) # pr/5 Phase 2: POST /api/v2/chat/stream (analysis_id)
|
| 72 |
|
| 73 |
|
|
|
|
| 22 |
from src.api.v1.tools import router as tools_router
|
| 23 |
from src.api.v1.help import router as help_router # pr/5 Phase 2: dedicated /tools/help
|
| 24 |
from src.api.v1.traceability import router as traceability_router # KM-691
|
| 25 |
+
from src.api.v1.charts import router as charts_router # W2: GET /api/v1/charts (SPINE_V2_PLAN §4.5)
|
| 26 |
from src.api.v2.chat import router as chat_v2_router # pr/5 Phase 2: v2 chat pilot (analysis_id)
|
| 27 |
from src.db.postgres.init_db import init_db
|
| 28 |
from src.config.settings import settings
|
|
|
|
| 69 |
app.include_router(tools_router)
|
| 70 |
app.include_router(help_router)
|
| 71 |
app.include_router(traceability_router) # KM-691: GET /api/v1/traceability
|
| 72 |
+
app.include_router(charts_router) # W2: GET /api/v1/charts (SPINE_V2_PLAN §4.5)
|
| 73 |
app.include_router(chat_v2_router) # pr/5 Phase 2: POST /api/v2/chat/stream (analysis_id)
|
| 74 |
|
| 75 |
|
src/agents/chat_handler.py
CHANGED
|
@@ -54,6 +54,7 @@ from .refusals import blocked_message, out_of_scope_message
|
|
| 54 |
|
| 55 |
if TYPE_CHECKING:
|
| 56 |
from ..catalog.reader import CatalogReader
|
|
|
|
| 57 |
from ..retrieval.router import RetrievalRouter
|
| 58 |
from ..traceability.store import TraceabilityStore
|
| 59 |
from .gate import AnalysisState
|
|
@@ -105,6 +106,7 @@ class ChatHandler:
|
|
| 105 |
) = None,
|
| 106 |
analysis_store: ReportInputStore | None = None,
|
| 107 |
traceability_store: TraceabilityStore | None = None,
|
|
|
|
| 108 |
check_invoker_factory: Callable[[str], Any] | None = None,
|
| 109 |
ps_agent: ProblemStatementAgent | None = None,
|
| 110 |
help_agent: HelpAgent | None = None,
|
|
@@ -128,6 +130,9 @@ class ChatHandler:
|
|
| 128 |
# Traceability (KM-691): user-facing per-turn provenance store. Injectable for
|
| 129 |
# tests; lazily built (Postgres) in production. Distinct from Langfuse tracing.
|
| 130 |
self._traceability_store = traceability_store
|
|
|
|
|
|
|
|
|
|
| 131 |
# `check` skill: builds the data-access invoker (check_data/check_knowledge)
|
| 132 |
# per request with the authenticated user_id. Injectable for tests.
|
| 133 |
self._check_invoker_factory = check_invoker_factory
|
|
@@ -644,6 +649,13 @@ class ChatHandler:
|
|
| 644 |
self._traceability_store = PostgresTraceabilityStore()
|
| 645 |
return self._traceability_store
|
| 646 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 647 |
async def _flush_trace(
|
| 648 |
self, pad: TraceabilityScratchpad, analysis_id: str | None, user_id: str
|
| 649 |
) -> None:
|
|
@@ -763,6 +775,22 @@ class ChatHandler:
|
|
| 763 |
pad.set_planning_from_record(record)
|
| 764 |
except Exception as e: # persistence must never break the user's answer
|
| 765 |
logger.error("analysis_record persist failed", user_id=user_id, error=str(e))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 766 |
tracer.end() # output omitted (chat_answer may contain PII on Cloud)
|
| 767 |
if pad is not None:
|
| 768 |
await self._flush_trace(pad, analysis_id, user_id)
|
|
|
|
| 54 |
|
| 55 |
if TYPE_CHECKING:
|
| 56 |
from ..catalog.reader import CatalogReader
|
| 57 |
+
from ..charts.store import ChartStore
|
| 58 |
from ..retrieval.router import RetrievalRouter
|
| 59 |
from ..traceability.store import TraceabilityStore
|
| 60 |
from .gate import AnalysisState
|
|
|
|
| 106 |
) = None,
|
| 107 |
analysis_store: ReportInputStore | None = None,
|
| 108 |
traceability_store: TraceabilityStore | None = None,
|
| 109 |
+
chart_store: ChartStore | None = None,
|
| 110 |
check_invoker_factory: Callable[[str], Any] | None = None,
|
| 111 |
ps_agent: ProblemStatementAgent | None = None,
|
| 112 |
help_agent: HelpAgent | None = None,
|
|
|
|
| 130 |
# Traceability (KM-691): user-facing per-turn provenance store. Injectable for
|
| 131 |
# tests; lazily built (Postgres) in production. Distinct from Langfuse tracing.
|
| 132 |
self._traceability_store = traceability_store
|
| 133 |
+
# Charts (S2, SPINE_V2_PLAN §4.4): `render_chart` output store. Injectable for
|
| 134 |
+
# tests; lazily built (Postgres) in production.
|
| 135 |
+
self._chart_store = chart_store
|
| 136 |
# `check` skill: builds the data-access invoker (check_data/check_knowledge)
|
| 137 |
# per request with the authenticated user_id. Injectable for tests.
|
| 138 |
self._check_invoker_factory = check_invoker_factory
|
|
|
|
| 649 |
self._traceability_store = PostgresTraceabilityStore()
|
| 650 |
return self._traceability_store
|
| 651 |
|
| 652 |
+
def _get_chart_store(self) -> ChartStore:
|
| 653 |
+
if self._chart_store is None:
|
| 654 |
+
from ..charts import PostgresChartStore
|
| 655 |
+
|
| 656 |
+
self._chart_store = PostgresChartStore()
|
| 657 |
+
return self._chart_store
|
| 658 |
+
|
| 659 |
async def _flush_trace(
|
| 660 |
self, pad: TraceabilityScratchpad, analysis_id: str | None, user_id: str
|
| 661 |
) -> None:
|
|
|
|
| 775 |
pad.set_planning_from_record(record)
|
| 776 |
except Exception as e: # persistence must never break the user's answer
|
| 777 |
logger.error("analysis_record persist failed", user_id=user_id, error=str(e))
|
| 778 |
+
# SPINE_V2_PLAN §4.4: chart rows are written before `done`; the FE fetches
|
| 779 |
+
# GET /api/v1/charts unconditionally on every `done` (no polling race).
|
| 780 |
+
try:
|
| 781 |
+
if pad is not None and pad.message_id:
|
| 782 |
+
for task_result in result.analysis_record.results_snapshot.values():
|
| 783 |
+
for output in task_result.outputs:
|
| 784 |
+
if output.kind == "chart" and isinstance(output.value, dict):
|
| 785 |
+
await self._get_chart_store().save(
|
| 786 |
+
message_id=pad.message_id,
|
| 787 |
+
analysis_id=analysis_id or "",
|
| 788 |
+
user_id=user_id,
|
| 789 |
+
record_id=result.analysis_record.record_id,
|
| 790 |
+
envelope=output.value,
|
| 791 |
+
)
|
| 792 |
+
except Exception as e: # chart persist must never break the user's answer
|
| 793 |
+
logger.error("chart persist failed", user_id=user_id, error=str(e))
|
| 794 |
tracer.end() # output omitted (chat_answer may contain PII on Cloud)
|
| 795 |
if pad is not None:
|
| 796 |
await self._flush_trace(pad, analysis_id, user_id)
|
src/agents/planner/examples.py
CHANGED
|
@@ -730,6 +730,119 @@ _EXAMPLE_I = TaskList(
|
|
| 730 |
)
|
| 731 |
|
| 732 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 733 |
EXAMPLES: list[tuple[str, TaskList]] = [
|
| 734 |
("Which product categories drove last quarter's revenue?", _EXAMPLE_A),
|
| 735 |
("How has monthly revenue trended by region this year, and what's unusual?", _EXAMPLE_B),
|
|
@@ -744,6 +857,8 @@ EXAMPLES: list[tuple[str, TaskList]] = [
|
|
| 744 |
"order quantity?",
|
| 745 |
_EXAMPLE_I,
|
| 746 |
),
|
|
|
|
|
|
|
| 747 |
]
|
| 748 |
|
| 749 |
|
|
|
|
| 730 |
)
|
| 731 |
|
| 732 |
|
| 733 |
+
# --------------------------------------------------------------------------- #
|
| 734 |
+
# Example J — visualization tail (SPINE_V2_PLAN §4.3, recipe "viz tail").
|
| 735 |
+
# "Show me a bar chart of total revenue per product category."
|
| 736 |
+
# Shows: render_chart is planned ONLY on an explicit ask ("bar chart"), and is
|
| 737 |
+
# ALWAYS a tail step — its `data` consumes a TABLE-producing upstream (here the
|
| 738 |
+
# grouped top-N-style retrieve from Example G, without the limit), never stats/
|
| 739 |
+
# series/metadata. `x`/`y` reference that table's column aliases; the chart
|
| 740 |
+
# carries the already-aggregated rows (one bar per category), never raw order
|
| 741 |
+
# rows. The `assumptions` line carries the feasibility check on purpose: a chart
|
| 742 |
+
# ask never licenses aliasing a stand-in column (see Example K).
|
| 743 |
+
# --------------------------------------------------------------------------- #
|
| 744 |
+
|
| 745 |
+
_EXAMPLE_J = TaskList(
|
| 746 |
+
plan_id="example_j",
|
| 747 |
+
goal_restated="Chart total revenue per product category as a bar chart.",
|
| 748 |
+
assumptions=[
|
| 749 |
+
"The chart dimension exists in the catalog (c_category) — a chart ask is "
|
| 750 |
+
"planned only when the asked-for dimension and measure have real catalog "
|
| 751 |
+
"columns; otherwise it is infeasible, chart or no chart."
|
| 752 |
+
],
|
| 753 |
+
open_questions=[],
|
| 754 |
+
tasks=[
|
| 755 |
+
Task(
|
| 756 |
+
id="t1",
|
| 757 |
+
stage="data_understanding",
|
| 758 |
+
objective="Confirm the sales source exposes category and revenue.",
|
| 759 |
+
tool_calls=[ToolCall(tool="check_data", args={"source_id": "src_sales"})],
|
| 760 |
+
expected_output="source_shape",
|
| 761 |
+
success_criteria="Produced the orders table schema; category and revenue present.",
|
| 762 |
+
depends_on=[],
|
| 763 |
+
estimated_cost="low",
|
| 764 |
+
),
|
| 765 |
+
Task(
|
| 766 |
+
id="t2",
|
| 767 |
+
stage="data_preparation",
|
| 768 |
+
objective="Aggregate total revenue per product category.",
|
| 769 |
+
tool_calls=[
|
| 770 |
+
ToolCall(
|
| 771 |
+
tool="retrieve_data",
|
| 772 |
+
args={
|
| 773 |
+
"ir": {
|
| 774 |
+
"source_id": "src_sales",
|
| 775 |
+
"table_id": "t_orders",
|
| 776 |
+
"select": [
|
| 777 |
+
{"kind": "column", "column_id": "c_category", "alias": "category"},
|
| 778 |
+
{
|
| 779 |
+
"kind": "agg",
|
| 780 |
+
"fn": "sum",
|
| 781 |
+
"column_id": "c_revenue",
|
| 782 |
+
"alias": "total_revenue",
|
| 783 |
+
},
|
| 784 |
+
],
|
| 785 |
+
"group_by": ["c_category"],
|
| 786 |
+
}
|
| 787 |
+
},
|
| 788 |
+
)
|
| 789 |
+
],
|
| 790 |
+
expected_output="revenue_per_category",
|
| 791 |
+
success_criteria="Produced one total-revenue row per category.",
|
| 792 |
+
depends_on=["t1"],
|
| 793 |
+
estimated_cost="low",
|
| 794 |
+
),
|
| 795 |
+
Task(
|
| 796 |
+
id="t3",
|
| 797 |
+
stage="evaluation",
|
| 798 |
+
objective="Render the per-category revenue table as a bar chart.",
|
| 799 |
+
tool_calls=[
|
| 800 |
+
ToolCall(
|
| 801 |
+
tool="render_chart",
|
| 802 |
+
args={
|
| 803 |
+
# `data` references the TABLE task (t2) — a chart is always a
|
| 804 |
+
# tail on an aggregated table, never on raw rows or stats.
|
| 805 |
+
"data": "${t2}",
|
| 806 |
+
"chart_type": "bar",
|
| 807 |
+
"x": "category",
|
| 808 |
+
"y": "total_revenue",
|
| 809 |
+
"title": "Total revenue by product category",
|
| 810 |
+
},
|
| 811 |
+
)
|
| 812 |
+
],
|
| 813 |
+
expected_output="revenue_bar_chart",
|
| 814 |
+
success_criteria="Produced a bar-chart spec with one bar per category.",
|
| 815 |
+
depends_on=["t2"],
|
| 816 |
+
estimated_cost="low",
|
| 817 |
+
),
|
| 818 |
+
],
|
| 819 |
+
)
|
| 820 |
+
|
| 821 |
+
|
| 822 |
+
# --------------------------------------------------------------------------- #
|
| 823 |
+
# Example K — a chart ask that is INFEASIBLE (SPINE_V2_PLAN §4.3 + planner.md
|
| 824 |
+
# "Charts ... never relaxes feasibility"). "Plot the customer churn rate by
|
| 825 |
+
# month as a line chart" against the sales catalog: churn does not exist, and
|
| 826 |
+
# the explicit chart request does NOT license mapping some other column into
|
| 827 |
+
# the asked-for measure — the verdict is the same infeasible_reason Example H
|
| 828 |
+
# would give, chart or no chart.
|
| 829 |
+
# --------------------------------------------------------------------------- #
|
| 830 |
+
|
| 831 |
+
_EXAMPLE_K = TaskList(
|
| 832 |
+
plan_id="example_k",
|
| 833 |
+
goal_restated="Chart the monthly customer churn rate as a line chart.",
|
| 834 |
+
assumptions=[],
|
| 835 |
+
open_questions=[],
|
| 836 |
+
tasks=[],
|
| 837 |
+
infeasible_reason=(
|
| 838 |
+
"The connected source has no churn or subscription-status data — the "
|
| 839 |
+
"orders table only carries order-level category, revenue, quantity, and "
|
| 840 |
+
"dates — so there is nothing to chart. Nearest chartable analyses: "
|
| 841 |
+
"monthly revenue trend, or order counts per category."
|
| 842 |
+
),
|
| 843 |
+
)
|
| 844 |
+
|
| 845 |
+
|
| 846 |
EXAMPLES: list[tuple[str, TaskList]] = [
|
| 847 |
("Which product categories drove last quarter's revenue?", _EXAMPLE_A),
|
| 848 |
("How has monthly revenue trended by region this year, and what's unusual?", _EXAMPLE_B),
|
|
|
|
| 857 |
"order quantity?",
|
| 858 |
_EXAMPLE_I,
|
| 859 |
),
|
| 860 |
+
("Show me a bar chart of total revenue per product category.", _EXAMPLE_J),
|
| 861 |
+
("Plot the customer churn rate by month as a line chart.", _EXAMPLE_K),
|
| 862 |
]
|
| 863 |
|
| 864 |
|
src/agents/planner/validator.py
CHANGED
|
@@ -281,6 +281,24 @@ class PlannerValidator:
|
|
| 281 |
"analyzable data rows. Feed analyze_* from a data-producing tool "
|
| 282 |
"(e.g. retrieve_data)."
|
| 283 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 284 |
|
| 285 |
@staticmethod
|
| 286 |
def _validate_dag(tasks_by_id: dict, id_set: set[str]) -> None:
|
|
|
|
| 281 |
"analyzable data rows. Feed analyze_* from a data-producing tool "
|
| 282 |
"(e.g. retrieve_data)."
|
| 283 |
)
|
| 284 |
+
# Check 10 — render_chart path shape (SPINE_V2_PLAN §4.3): the chart's
|
| 285 |
+
# `data` must come from a task whose tool yields a TABLE. A stats- or
|
| 286 |
+
# series-kind upstream passes the category denylist above but cannot
|
| 287 |
+
# be materialized into a DataFrame at execution, so the chart could
|
| 288 |
+
# only fail late; reject pre-run so the retry re-prompts toward a
|
| 289 |
+
# grouped retrieve_data or a table-producing analyze_*.
|
| 290 |
+
if (
|
| 291 |
+
call.tool == "render_chart"
|
| 292 |
+
and ref_spec is not None
|
| 293 |
+
and ref_spec.output_kind != "table"
|
| 294 |
+
):
|
| 295 |
+
raise PlannerValidationError(
|
| 296 |
+
f"task {task_id}: render_chart takes its {arg_name!r} from task "
|
| 297 |
+
f"{match.group(1)} ({ref_tool!r}), which yields "
|
| 298 |
+
f"{ref_spec.output_kind!r} output — a chart is drawn from a TABLE. "
|
| 299 |
+
"Feed it a table-producing task (a grouped retrieve_data, "
|
| 300 |
+
"analyze_aggregate, analyze_merge, ...)."
|
| 301 |
+
)
|
| 302 |
|
| 303 |
@staticmethod
|
| 304 |
def _validate_dag(tasks_by_id: dict, id_set: set[str]) -> None:
|
src/api/v1/charts.py
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Charts endpoint — user-facing chart retrieval for one turn (S2 visualization,
|
| 2 |
+
SPINE_V2_PLAN §4.4/§4.5).
|
| 3 |
+
|
| 4 |
+
`GET /api/v1/charts?analysis_id=&message_id=` returns every chart a `render_chart`
|
| 5 |
+
tool call produced during one assistant turn, as `dataeyond.chart.v1` envelopes
|
| 6 |
+
(SPINE_V2_PLAN §4.2). Rows are written by the chat pipeline right before the `done`
|
| 7 |
+
SSE event; the FE fires this GET on `done`.
|
| 8 |
+
|
| 9 |
+
Unlike `GET /api/v1/traceability`, an **empty list is a valid 200** here (most turns
|
| 10 |
+
don't ask for a chart) rather than a 404 — once `done` has fired there is no "not
|
| 11 |
+
ready yet" state to distinguish from "no charts this turn".
|
| 12 |
+
|
| 13 |
+
No auth — Go fronts Python.
|
| 14 |
+
"""
|
| 15 |
+
|
| 16 |
+
from fastapi import APIRouter, Query
|
| 17 |
+
from pydantic import BaseModel
|
| 18 |
+
|
| 19 |
+
from src.charts import ChartRecord, PostgresChartStore
|
| 20 |
+
from src.middlewares.logging import get_logger, log_execution
|
| 21 |
+
|
| 22 |
+
logger = get_logger("charts_api")
|
| 23 |
+
|
| 24 |
+
router = APIRouter(prefix="/api/v1", tags=["Charts"])
|
| 25 |
+
|
| 26 |
+
# Warm, process-shared store (mirrors the chat endpoints' module-level instances).
|
| 27 |
+
_store = PostgresChartStore()
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
class ChartsResponse(BaseModel):
|
| 31 |
+
count: int
|
| 32 |
+
charts: list[ChartRecord]
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
@router.get("/charts", response_model=ChartsResponse)
|
| 36 |
+
@log_execution(logger)
|
| 37 |
+
async def get_charts(
|
| 38 |
+
analysis_id: str = Query(..., description="Analysis/session id"),
|
| 39 |
+
message_id: str = Query(
|
| 40 |
+
..., description="Assistant turn id, taken from the `done` SSE event"
|
| 41 |
+
),
|
| 42 |
+
) -> ChartsResponse:
|
| 43 |
+
"""Fetch every chart for one turn. Empty list (not 404) when the turn produced
|
| 44 |
+
none — the FE can call this unconditionally on every `done`."""
|
| 45 |
+
charts = await _store.list_for_message(analysis_id, message_id)
|
| 46 |
+
return ChartsResponse(count=len(charts), charts=charts)
|
src/charts/__init__.py
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Charts (S2 visualization, SPINE_V2_PLAN §4.4): `render_chart` output persistence.
|
| 2 |
+
|
| 3 |
+
Chart envelopes produced by the `render_chart` tool (`dataeyond.chart.v1`, SPINE_V2_PLAN
|
| 4 |
+
§4.2) are persisted through a `ChartStore` right before the `done` SSE event, and served
|
| 5 |
+
by `GET /api/v1/charts`. SSE stays text-only — charts are never embedded in the streamed
|
| 6 |
+
answer.
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
from .store import (
|
| 10 |
+
ChartRecord,
|
| 11 |
+
ChartStore,
|
| 12 |
+
NullChartStore,
|
| 13 |
+
PostgresChartStore,
|
| 14 |
+
)
|
| 15 |
+
|
| 16 |
+
__all__ = [
|
| 17 |
+
"ChartRecord",
|
| 18 |
+
"ChartStore",
|
| 19 |
+
"NullChartStore",
|
| 20 |
+
"PostgresChartStore",
|
| 21 |
+
]
|
src/charts/store.py
ADDED
|
@@ -0,0 +1,154 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""ChartStore — the seam the chat pipeline persists `render_chart` outputs through
|
| 2 |
+
(S2 visualization, SPINE_V2_PLAN §4.4).
|
| 3 |
+
|
| 4 |
+
`ChatHandler._run_slow_path` scans a completed `AnalysisRecord.results_snapshot` for
|
| 5 |
+
`ToolOutput(kind="chart")` entries and flushes one `MessageChartRow` per chart through
|
| 6 |
+
this seam, right before the `done` SSE event (mirrors the traceability flush). Unlike
|
| 7 |
+
traceability's one row per turn, a turn with multiple `render_chart` calls writes
|
| 8 |
+
multiple rows sharing one `message_id`. `GET /api/v1/charts` reads them back by
|
| 9 |
+
(analysis_id, message_id).
|
| 10 |
+
|
| 11 |
+
- `NullChartStore` logs the envelope and stores nothing (tests / disabled persistence).
|
| 12 |
+
- `PostgresChartStore` writes one `message_charts` row per chart (dedorch,
|
| 13 |
+
`AsyncSessionLocal`), mirroring `PostgresTraceabilityStore`.
|
| 14 |
+
|
| 15 |
+
`save` must NEVER raise on the caller's path — a chart-persist failure must not break
|
| 16 |
+
the user's answer. `list_for_message` is the endpoint read; an empty list is a valid
|
| 17 |
+
result (chartless turn), not an error.
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
from __future__ import annotations
|
| 21 |
+
|
| 22 |
+
import uuid
|
| 23 |
+
from datetime import datetime
|
| 24 |
+
from typing import Protocol, runtime_checkable
|
| 25 |
+
|
| 26 |
+
from pydantic import BaseModel
|
| 27 |
+
from sqlalchemy import select
|
| 28 |
+
|
| 29 |
+
from src.db.postgres.connection import AsyncSessionLocal
|
| 30 |
+
from src.db.postgres.models import MessageChartRow
|
| 31 |
+
from src.middlewares.logging import get_logger
|
| 32 |
+
|
| 33 |
+
logger = get_logger("charts_store")
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
class ChartRecord(BaseModel):
|
| 37 |
+
"""One persisted chart, as served by `GET /api/v1/charts`."""
|
| 38 |
+
|
| 39 |
+
chart_id: str
|
| 40 |
+
chart_type: str
|
| 41 |
+
title: str | None = None
|
| 42 |
+
spec: dict
|
| 43 |
+
created_at: datetime
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
@runtime_checkable
|
| 47 |
+
class ChartStore(Protocol):
|
| 48 |
+
"""Persist + read `render_chart` outputs for one assistant `message_id`.
|
| 49 |
+
|
| 50 |
+
`save` must never raise on the caller's path. `list_for_message` returns every
|
| 51 |
+
chart for one turn (possibly empty — a turn need not have asked for a chart).
|
| 52 |
+
"""
|
| 53 |
+
|
| 54 |
+
async def save(
|
| 55 |
+
self,
|
| 56 |
+
*,
|
| 57 |
+
message_id: str,
|
| 58 |
+
analysis_id: str,
|
| 59 |
+
user_id: str,
|
| 60 |
+
record_id: str | None,
|
| 61 |
+
envelope: dict,
|
| 62 |
+
) -> None: ...
|
| 63 |
+
|
| 64 |
+
async def list_for_message(
|
| 65 |
+
self, analysis_id: str, message_id: str
|
| 66 |
+
) -> list[ChartRecord]: ...
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
class NullChartStore:
|
| 70 |
+
"""No-op store: logs the envelope, persists nothing. Reads return `[]`."""
|
| 71 |
+
|
| 72 |
+
async def save(
|
| 73 |
+
self,
|
| 74 |
+
*,
|
| 75 |
+
message_id: str,
|
| 76 |
+
analysis_id: str,
|
| 77 |
+
user_id: str,
|
| 78 |
+
record_id: str | None,
|
| 79 |
+
envelope: dict,
|
| 80 |
+
) -> None:
|
| 81 |
+
logger.info(
|
| 82 |
+
"chart produced (not persisted — NullChartStore)",
|
| 83 |
+
message_id=message_id,
|
| 84 |
+
chart_type=envelope.get("chart_type", "unknown"),
|
| 85 |
+
)
|
| 86 |
+
|
| 87 |
+
async def list_for_message(
|
| 88 |
+
self, analysis_id: str, message_id: str
|
| 89 |
+
) -> list[ChartRecord]:
|
| 90 |
+
return []
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
class PostgresChartStore:
|
| 94 |
+
"""Writes/reads `message_charts` rows. One insert per chart (not an upsert)."""
|
| 95 |
+
|
| 96 |
+
async def save(
|
| 97 |
+
self,
|
| 98 |
+
*,
|
| 99 |
+
message_id: str,
|
| 100 |
+
analysis_id: str,
|
| 101 |
+
user_id: str,
|
| 102 |
+
record_id: str | None,
|
| 103 |
+
envelope: dict,
|
| 104 |
+
) -> None:
|
| 105 |
+
try:
|
| 106 |
+
async with AsyncSessionLocal() as session:
|
| 107 |
+
row = MessageChartRow(
|
| 108 |
+
id=str(uuid.uuid4()),
|
| 109 |
+
message_id=message_id,
|
| 110 |
+
analysis_id=analysis_id,
|
| 111 |
+
user_id=user_id,
|
| 112 |
+
record_id=record_id,
|
| 113 |
+
chart_type=envelope.get("chart_type", "unknown"),
|
| 114 |
+
title=envelope.get("title"),
|
| 115 |
+
spec=envelope,
|
| 116 |
+
)
|
| 117 |
+
session.add(row)
|
| 118 |
+
await session.commit()
|
| 119 |
+
logger.info(
|
| 120 |
+
"chart persisted",
|
| 121 |
+
message_id=message_id,
|
| 122 |
+
analysis_id=analysis_id,
|
| 123 |
+
chart_type=envelope.get("chart_type", "unknown"),
|
| 124 |
+
)
|
| 125 |
+
except Exception as exc: # never break the user's answer
|
| 126 |
+
logger.error(
|
| 127 |
+
"chart persist failed",
|
| 128 |
+
message_id=message_id,
|
| 129 |
+
error=str(exc),
|
| 130 |
+
)
|
| 131 |
+
|
| 132 |
+
async def list_for_message(
|
| 133 |
+
self, analysis_id: str, message_id: str
|
| 134 |
+
) -> list[ChartRecord]:
|
| 135 |
+
async with AsyncSessionLocal() as session:
|
| 136 |
+
result = await session.execute(
|
| 137 |
+
select(MessageChartRow)
|
| 138 |
+
.where(
|
| 139 |
+
MessageChartRow.analysis_id == analysis_id,
|
| 140 |
+
MessageChartRow.message_id == message_id,
|
| 141 |
+
)
|
| 142 |
+
.order_by(MessageChartRow.created_at)
|
| 143 |
+
)
|
| 144 |
+
rows = result.scalars().all()
|
| 145 |
+
return [
|
| 146 |
+
ChartRecord(
|
| 147 |
+
chart_id=row.id,
|
| 148 |
+
chart_type=row.chart_type,
|
| 149 |
+
title=row.title,
|
| 150 |
+
spec=row.spec,
|
| 151 |
+
created_at=row.created_at,
|
| 152 |
+
)
|
| 153 |
+
for row in rows
|
| 154 |
+
]
|
src/config/prompts/planner.md
CHANGED
|
@@ -20,7 +20,9 @@ only a `TaskList` object that conforms to the provided schema.
|
|
| 20 |
section. Copy the stable ids verbatim — downstream validation does a literal
|
| 21 |
id lookup, so a paraphrased name fails.
|
| 22 |
5. **No modeling in v1.** There are no modeling tools. Do not emit `modeling`
|
| 23 |
-
tasks. The product is descriptive/diagnostic only — no predictions
|
|
|
|
|
|
|
| 24 |
6. **Never re-purpose a column as a different business measure.** A column means
|
| 25 |
what the catalog says it means — do not alias one concept as another to force
|
| 26 |
an answer (e.g. selecting an availability percentage AS "revenue", or a
|
|
@@ -35,6 +37,21 @@ only a `TaskList` object that conforms to the provided schema.
|
|
| 35 |
is noise, not helpfulness. A multi-part task list is correct ONLY when the
|
| 36 |
question itself has multiple parts (e.g. "trend by region AND what's unusual").
|
| 37 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
# How to plan
|
| 39 |
|
| 40 |
- **Smallest plan that answers the question.** Do not exceed `Constraints.max_tasks`.
|
|
@@ -82,6 +99,20 @@ only a `TaskList` object that conforms to the provided schema.
|
|
| 82 |
to a task's LAST output, so two retrievals inside one task lose the first
|
| 83 |
table. The merged table (one row per entity, both measures) answers the
|
| 84 |
question, or feeds a further `analyze_*` step.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
- **Mixing structured + unstructured.** If qualitative context helps, add a
|
| 86 |
`retrieve_knowledge` task against an unstructured source listed in the catalog.
|
| 87 |
- **CRISP-DM stages.** Tag each task with the stage it serves:
|
|
|
|
| 20 |
section. Copy the stable ids verbatim — downstream validation does a literal
|
| 21 |
id lookup, so a paraphrased name fails.
|
| 22 |
5. **No modeling in v1.** There are no modeling tools. Do not emit `modeling`
|
| 23 |
+
tasks. The product is descriptive/diagnostic only — no predictions. A chart
|
| 24 |
+
exists ONLY as a `render_chart` tail on an explicit user ask (see "Charts"
|
| 25 |
+
under "How to plan") — never speculatively.
|
| 26 |
6. **Never re-purpose a column as a different business measure.** A column means
|
| 27 |
what the catalog says it means — do not alias one concept as another to force
|
| 28 |
an answer (e.g. selecting an availability percentage AS "revenue", or a
|
|
|
|
| 37 |
is noise, not helpfulness. A multi-part task list is correct ONLY when the
|
| 38 |
question itself has multiple parts (e.g. "trend by region AND what's unusual").
|
| 39 |
|
| 40 |
+
# Recipes — the named workflows
|
| 41 |
+
|
| 42 |
+
Classify the question into ONE recipe, then instantiate it. Most questions are a
|
| 43 |
+
recipe verbatim; a genuinely multi-part question composes recipes.
|
| 44 |
+
|
| 45 |
+
| Recipe | The question asks for… | Chain |
|
| 46 |
+
|---|---|---|
|
| 47 |
+
| R1 descriptive | a summary/distribution of columns | `retrieve_data` → `analyze_descriptive` |
|
| 48 |
+
| R2 aggregate / top-N | totals or averages per group, "top N by …" | ONE grouped `retrieve_data` IR (± `analyze_aggregate`) |
|
| 49 |
+
| R3 trend | movement over time | `retrieve_data` → `analyze_trend` |
|
| 50 |
+
| R4 correlation | the relationship between numeric columns | `retrieve_data` → `analyze_correlation` |
|
| 51 |
+
| R5 two-metric merge | "which X has both A and B" | `retrieve_data` ×2 → `analyze_merge` → … |
|
| 52 |
+
| R6 infeasible | a measure/entity no catalog column holds | `tasks: []` + `infeasible_reason` |
|
| 53 |
+
| viz tail | …any of the above, WITH an explicit ask to chart it | R1–R5 + a `render_chart` tail |
|
| 54 |
+
|
| 55 |
# How to plan
|
| 56 |
|
| 57 |
- **Smallest plan that answers the question.** Do not exceed `Constraints.max_tasks`.
|
|
|
|
| 99 |
to a task's LAST output, so two retrievals inside one task lose the first
|
| 100 |
table. The merged table (one row per entity, both measures) answers the
|
| 101 |
question, or feeds a further `analyze_*` step.
|
| 102 |
+
- **Charts (`render_chart`) only on an explicit ask.** Add a `render_chart` tail
|
| 103 |
+
ONLY when the user explicitly asks to see a chart — "plot", "chart", "graph",
|
| 104 |
+
"visualize", "diagram" (ID: "grafik", "diagram", "visualisasikan", "buatkan
|
| 105 |
+
grafik/diagram") — NEVER speculatively on a plain data question. It is always
|
| 106 |
+
the LAST step: its `data` takes a `"${t<id>}"` from a task that yields a
|
| 107 |
+
**table** (a grouped `retrieve_data` or a table-producing `analyze_*` — never
|
| 108 |
+
stats/series/metadata), and `x`/`y` are that table's column aliases. Chart the
|
| 109 |
+
already-aggregated table (one row per category/period), not raw rows. Pick
|
| 110 |
+
`chart_type` by the question: `bar` (magnitude per category), `line` (over
|
| 111 |
+
time), `pie` (share of a small whole), `scatter` (two numeric columns).
|
| 112 |
+
A chart ask NEVER relaxes feasibility (rule 6): if the asked-for dimension or
|
| 113 |
+
measure has no catalog column, the question is **infeasible** — never chart a
|
| 114 |
+
stand-in column aliased under the asked-for name (e.g. never select a status
|
| 115 |
+
column AS "region" because the user asked for a chart by region).
|
| 116 |
- **Mixing structured + unstructured.** If qualitative context helps, add a
|
| 117 |
`retrieve_knowledge` task against an unstructured source listed in the catalog.
|
| 118 |
- **CRISP-DM stages.** Tag each task with the stage it serves:
|
src/db/postgres/models.py
CHANGED
|
@@ -284,3 +284,36 @@ class MessageTraceabilityRow(Base):
|
|
| 284 |
intent = Column(String, nullable=False)
|
| 285 |
data = Column(JSONB, nullable=False) # full TraceabilityPayload (source of truth)
|
| 286 |
created_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 284 |
intent = Column(String, nullable=False)
|
| 285 |
data = Column(JSONB, nullable=False) # full TraceabilityPayload (source of truth)
|
| 286 |
created_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
| 287 |
+
|
| 288 |
+
|
| 289 |
+
class MessageChartRow(Base):
|
| 290 |
+
"""One row per chart produced by `render_chart` — S2 visualization (SPINE_V2_PLAN §4.4).
|
| 291 |
+
|
| 292 |
+
`spec` holds the full `dataeyond.chart.v1` envelope (SPINE_V2_PLAN §4.2) exactly as
|
| 293 |
+
returned by the tool — `{schema, chart_type, title, plotly: {data, layout}}` — it is
|
| 294 |
+
the source of truth the FE renders with
|
| 295 |
+
`Plotly.newPlot(el, spec.plotly.data, spec.plotly.layout)`. Unlike
|
| 296 |
+
`MessageTraceabilityRow` (one row per turn), **multiple rows can share one
|
| 297 |
+
`message_id`** — one row per chart a turn produced. Written before the `done` SSE
|
| 298 |
+
event and served by `GET /api/v1/charts` (SPINE_V2_PLAN §4.4/§4.5).
|
| 299 |
+
|
| 300 |
+
OWNERSHIP / HANDOFF (SPINE_V2_PLAN §4.4, 2026-07-13): **Python-owned for now**, the
|
| 301 |
+
same pattern as `report_inputs` / `message_traceability`. Post-cutover `init_db` no
|
| 302 |
+
longer runs `create_all`, so the table is created by a one-time manual DDL run
|
| 303 |
+
against dedorch (Rifqi, 2026-07-13); the finalized schema goes to Harry so the
|
| 304 |
+
dedorch migration re-creates it later (`analysis_id` gains the FK to `analyses(id)`
|
| 305 |
+
there).
|
| 306 |
+
"""
|
| 307 |
+
__tablename__ = "message_charts"
|
| 308 |
+
|
| 309 |
+
# Client-minted default — the DDL's `gen_random_uuid()` default never fires from
|
| 310 |
+
# Python (we always pass `id` explicitly, but the default is here for parity).
|
| 311 |
+
id = Column(UUID(as_uuid=False), primary_key=True, default=lambda: str(uuid4()))
|
| 312 |
+
message_id = Column(String, nullable=False, index=True) # turn id; multiple rows per turn
|
| 313 |
+
analysis_id = Column(UUID(as_uuid=False), nullable=False, index=True) # analysis session id
|
| 314 |
+
user_id = Column(String, nullable=False)
|
| 315 |
+
record_id = Column(String, nullable=True)
|
| 316 |
+
chart_type = Column(String, nullable=False)
|
| 317 |
+
title = Column(String, nullable=True)
|
| 318 |
+
spec = Column(JSONB, nullable=False) # full dataeyond.chart.v1 envelope (source of truth)
|
| 319 |
+
created_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
|
src/tools/analytics/visualization.py
ADDED
|
@@ -0,0 +1,239 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""render_chart — declarative chart-spec builder (SPINE_V2_PLAN §4.1, S2).
|
| 2 |
+
|
| 3 |
+
First tool of the `render_*` family: turns an already-materialized DataFrame into
|
| 4 |
+
a Plotly-conformant JSON spec wrapped in the `dataeyond.chart.v1` envelope
|
| 5 |
+
(SPINE_V2_PLAN §4.2). The FE renders it with plotly.js
|
| 6 |
+
(`Plotly.newPlot(el, spec.plotly.data, spec.plotly.layout)`).
|
| 7 |
+
|
| 8 |
+
Deliberately NOT a code generator and NOT a plotly import (locked decision,
|
| 9 |
+
DEV_PLAN deferred row #26): the spec is a hand-built dict, so no generated code
|
| 10 |
+
ever executes and no new dependency lands. Style (colors, axis chrome) is the
|
| 11 |
+
fixed module preset below — never a planner/LLM decision (CoDA's "Design
|
| 12 |
+
Explorer" phase collapsed to a constant).
|
| 13 |
+
|
| 14 |
+
Pattern A, same as the `analyze_*` family: `data` is resolved to a DataFrame by
|
| 15 |
+
the invoker from an upstream table-kind output; this function never fetches.
|
| 16 |
+
"""
|
| 17 |
+
|
| 18 |
+
from __future__ import annotations
|
| 19 |
+
|
| 20 |
+
import datetime as _dt
|
| 21 |
+
from decimal import Decimal
|
| 22 |
+
|
| 23 |
+
import numpy as np
|
| 24 |
+
import pandas as pd
|
| 25 |
+
|
| 26 |
+
from src.tools.analytics.descriptive import ColumnNotFoundError
|
| 27 |
+
|
| 28 |
+
# v1 chart types (SPINE_V2_PLAN §4.1). `pie` maps x -> labels, y -> values.
|
| 29 |
+
CHART_TYPES = ("bar", "line", "pie", "scatter")
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
class UnsupportedChartTypeError(ValueError):
|
| 33 |
+
"""The requested chart_type is not in CHART_TYPES (error_code UNSUPPORTED_CHART_TYPE)."""
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
# --------------------------------------------------------------------------- #
|
| 37 |
+
# House style preset — fixed by design, not a planner argument.
|
| 38 |
+
#
|
| 39 |
+
# Colorway = the dataviz reference categorical palette (8 slots, light mode).
|
| 40 |
+
# The slot ORDER is the colorblind-safety mechanism (validated: worst adjacent
|
| 41 |
+
# CVD deltaE 24.2 vs the >=12 target) — series take slots in this fixed order,
|
| 42 |
+
# never cycled or re-picked. Backgrounds are transparent so the chart inherits
|
| 43 |
+
# the FE surface; inks/gridlines are the palette's chrome roles.
|
| 44 |
+
# --------------------------------------------------------------------------- #
|
| 45 |
+
COLORWAY = [
|
| 46 |
+
"#2a78d6", # blue
|
| 47 |
+
"#1baf7a", # aqua
|
| 48 |
+
"#eda100", # yellow
|
| 49 |
+
"#008300", # green
|
| 50 |
+
"#4a3aa7", # violet
|
| 51 |
+
"#e34948", # red
|
| 52 |
+
"#e87ba4", # magenta
|
| 53 |
+
"#eb6834", # orange
|
| 54 |
+
]
|
| 55 |
+
|
| 56 |
+
_INK_SECONDARY = "#52514e" # body text
|
| 57 |
+
_INK_MUTED = "#898781" # axis titles / tick labels (mode-neutral)
|
| 58 |
+
_GRIDLINE = "#e1e0d9" # hairline grid
|
| 59 |
+
_BASELINE = "#c3c2b7" # axis line
|
| 60 |
+
|
| 61 |
+
LAYOUT_PRESET: dict[str, object] = {
|
| 62 |
+
"colorway": COLORWAY,
|
| 63 |
+
"font": {
|
| 64 |
+
"family": 'system-ui, -apple-system, "Segoe UI", sans-serif',
|
| 65 |
+
"size": 13,
|
| 66 |
+
"color": _INK_SECONDARY,
|
| 67 |
+
},
|
| 68 |
+
"paper_bgcolor": "rgba(0,0,0,0)",
|
| 69 |
+
"plot_bgcolor": "rgba(0,0,0,0)",
|
| 70 |
+
"margin": {"t": 48, "r": 16, "b": 48, "l": 56},
|
| 71 |
+
"bargap": 0.25, # keeps adjacent bars visually separated (mark-spec gap)
|
| 72 |
+
}
|
| 73 |
+
|
| 74 |
+
# Mark specs: thin marks — 2px lines, 8px markers.
|
| 75 |
+
_LINE_STYLE = {"width": 2}
|
| 76 |
+
_MARKER_STYLE = {"size": 8}
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def _axis(label: str) -> dict[str, object]:
|
| 80 |
+
"""Recessive axis chrome + the column name as the axis title."""
|
| 81 |
+
return {
|
| 82 |
+
"title": {"text": label, "font": {"color": _INK_MUTED}},
|
| 83 |
+
"tickfont": {"color": _INK_MUTED},
|
| 84 |
+
"gridcolor": _GRIDLINE,
|
| 85 |
+
"linecolor": _BASELINE,
|
| 86 |
+
"automargin": True,
|
| 87 |
+
}
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
def _clean(value: object) -> object:
|
| 91 |
+
"""One JSON-safe scalar: numpy -> Python, NaN/NaT -> None, dates -> ISO strings.
|
| 92 |
+
|
| 93 |
+
Chart specs are persisted as JSONB and shipped to the FE, so every value in
|
| 94 |
+
the trace arrays must survive json.dumps. Upstream `_materialize` already
|
| 95 |
+
normalizes fully-numeric columns, but mixed columns can still carry Decimal,
|
| 96 |
+
and date columns carry Timestamps.
|
| 97 |
+
"""
|
| 98 |
+
if value is None or value is pd.NaT:
|
| 99 |
+
return None
|
| 100 |
+
if isinstance(value, float) and np.isnan(value):
|
| 101 |
+
return None
|
| 102 |
+
if hasattr(value, "item"): # numpy scalar (incl. np.datetime64 via Timestamp below)
|
| 103 |
+
value = value.item()
|
| 104 |
+
if isinstance(value, float) and np.isnan(value):
|
| 105 |
+
return None
|
| 106 |
+
if isinstance(value, Decimal):
|
| 107 |
+
return float(value)
|
| 108 |
+
if isinstance(value, pd.Timestamp):
|
| 109 |
+
return value.isoformat()
|
| 110 |
+
if isinstance(value, _dt.datetime | _dt.date):
|
| 111 |
+
return value.isoformat()
|
| 112 |
+
if isinstance(value, bool | int | float | str):
|
| 113 |
+
return value
|
| 114 |
+
return str(value)
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
def _values(col: pd.Series) -> list[object]:
|
| 118 |
+
return [_clean(v) for v in col.tolist()]
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
def _series_label(value: object) -> str:
|
| 122 |
+
"""Trace name for one series group; null groups get an explicit label."""
|
| 123 |
+
cleaned = _clean(value)
|
| 124 |
+
return "(missing)" if cleaned is None else str(cleaned)
|
| 125 |
+
|
| 126 |
+
|
| 127 |
+
# Prompt-style description read by the Planner to decide WHEN to pick this tool.
|
| 128 |
+
DESCRIPTION = """\
|
| 129 |
+
Summary: Build a chart specification (bar, line, pie, scatter) from an \
|
| 130 |
+
already-retrieved table, for the app to render. Maps one x column and one y \
|
| 131 |
+
column (pie: x -> slice labels, y -> slice values); an optional `series` column \
|
| 132 |
+
splits bar/line/scatter into one trace per category. Produces a spec only — it \
|
| 133 |
+
computes no numbers.
|
| 134 |
+
|
| 135 |
+
USE WHEN the user EXPLICITLY asks to see a chart — "plot", "chart", "graph", \
|
| 136 |
+
"visualize", "diagram" (ID: "grafik", "diagram", "plot", "visualisasikan", \
|
| 137 |
+
"buatkan grafik/diagram"). ALWAYS a tail step: `data` consumes the output of an \
|
| 138 |
+
upstream task that yields a table (retrieve_data or a table-kind analyze_*).
|
| 139 |
+
|
| 140 |
+
DON'T USE WHEN:
|
| 141 |
+
- the user did not explicitly ask for a chart -> answer with tables/stats only \
|
| 142 |
+
(never add a speculative chart)
|
| 143 |
+
- the numbers still need computing -> run the analyze_*/retrieve step first, \
|
| 144 |
+
then chart its table output
|
| 145 |
+
- the upstream output is stats- or series-kind -> feed it a table-kind task \
|
| 146 |
+
(e.g. a grouped retrieve or analyze_aggregate)
|
| 147 |
+
|
| 148 |
+
Example questions:
|
| 149 |
+
- "plot revenue by region as a bar chart"
|
| 150 |
+
- "buatkan grafik penjualan per kategori"
|
| 151 |
+
- "show a pie chart of orders by sales channel"
|
| 152 |
+
- "visualize monthly revenue" (chain a month-grouped table first, then line)
|
| 153 |
+
"""
|
| 154 |
+
|
| 155 |
+
|
| 156 |
+
def render_chart(
|
| 157 |
+
df: pd.DataFrame,
|
| 158 |
+
chart_type: str,
|
| 159 |
+
x: str,
|
| 160 |
+
y: str,
|
| 161 |
+
series: str | None = None,
|
| 162 |
+
title: str | None = None,
|
| 163 |
+
) -> dict[str, object]:
|
| 164 |
+
"""Build a `dataeyond.chart.v1` envelope from a table (SPINE_V2_PLAN §4.2).
|
| 165 |
+
|
| 166 |
+
Args:
|
| 167 |
+
df: already-materialized data (the invoker resolves `data` upstream).
|
| 168 |
+
x: column for the x axis (pie: slice labels).
|
| 169 |
+
y: column for the y axis (pie: slice values).
|
| 170 |
+
series: optional column splitting bar/line/scatter into one trace per
|
| 171 |
+
distinct value (fixed-order colorway slots). Ignored for pie.
|
| 172 |
+
title: chart title; defaults to "<y> by <x>".
|
| 173 |
+
|
| 174 |
+
Returns:
|
| 175 |
+
The envelope dict: {schema, chart_type, title, plotly: {data, layout}}.
|
| 176 |
+
|
| 177 |
+
Raises:
|
| 178 |
+
UnsupportedChartTypeError: chart_type not in CHART_TYPES.
|
| 179 |
+
ColumnNotFoundError: x, y, or series absent from the data.
|
| 180 |
+
"""
|
| 181 |
+
if chart_type not in CHART_TYPES:
|
| 182 |
+
raise UnsupportedChartTypeError(
|
| 183 |
+
f"unsupported chart_type '{chart_type}'; supported: {list(CHART_TYPES)}"
|
| 184 |
+
)
|
| 185 |
+
needed = [x, y] if chart_type == "pie" or series is None else [x, y, series]
|
| 186 |
+
missing = [c for c in needed if c not in df.columns]
|
| 187 |
+
if missing:
|
| 188 |
+
raise ColumnNotFoundError(f"columns not found: {missing}")
|
| 189 |
+
|
| 190 |
+
chart_title = title or f"{y} by {x}"
|
| 191 |
+
|
| 192 |
+
if chart_type == "pie":
|
| 193 |
+
traces: list[dict[str, object]] = [
|
| 194 |
+
{"type": "pie", "labels": _values(df[x]), "values": _values(df[y])}
|
| 195 |
+
]
|
| 196 |
+
else:
|
| 197 |
+
plotly_type = "bar" if chart_type == "bar" else "scatter"
|
| 198 |
+
mode = {"line": "lines+markers", "scatter": "markers"}.get(chart_type)
|
| 199 |
+
|
| 200 |
+
def _trace(frame: pd.DataFrame, name: str) -> dict[str, object]:
|
| 201 |
+
t: dict[str, object] = {
|
| 202 |
+
"type": plotly_type,
|
| 203 |
+
"x": _values(frame[x]),
|
| 204 |
+
"y": _values(frame[y]),
|
| 205 |
+
"name": name,
|
| 206 |
+
}
|
| 207 |
+
if mode is not None:
|
| 208 |
+
t["mode"] = mode
|
| 209 |
+
t["line"] = dict(_LINE_STYLE)
|
| 210 |
+
t["marker"] = dict(_MARKER_STYLE)
|
| 211 |
+
return t
|
| 212 |
+
|
| 213 |
+
if series is None:
|
| 214 |
+
traces = [_trace(df, y)]
|
| 215 |
+
else:
|
| 216 |
+
# sort=True keeps trace order (and therefore colorway slot
|
| 217 |
+
# assignment) deterministic across runs of the same data.
|
| 218 |
+
traces = [
|
| 219 |
+
_trace(group, _series_label(value))
|
| 220 |
+
for value, group in df.groupby(series, dropna=False, sort=True)
|
| 221 |
+
]
|
| 222 |
+
|
| 223 |
+
layout: dict[str, object] = {
|
| 224 |
+
**LAYOUT_PRESET,
|
| 225 |
+
"title": {"text": chart_title},
|
| 226 |
+
# Legend only when there is more than one thing to identify: multi-series
|
| 227 |
+
# traces, or pie slices (a single named series is titled, not legended).
|
| 228 |
+
"showlegend": len(traces) > 1 or chart_type == "pie",
|
| 229 |
+
}
|
| 230 |
+
if chart_type != "pie":
|
| 231 |
+
layout["xaxis"] = _axis(x)
|
| 232 |
+
layout["yaxis"] = _axis(y)
|
| 233 |
+
|
| 234 |
+
return {
|
| 235 |
+
"schema": "dataeyond.chart.v1",
|
| 236 |
+
"chart_type": chart_type,
|
| 237 |
+
"title": chart_title,
|
| 238 |
+
"plotly": {"data": traces, "layout": layout},
|
| 239 |
+
}
|
src/tools/contracts.py
CHANGED
|
@@ -68,7 +68,7 @@ class ToolRegistry(BaseModel):
|
|
| 68 |
|
| 69 |
class ToolOutput(BaseModel):
|
| 70 |
tool: str
|
| 71 |
-
kind: Literal["scalar", "table", "stats", "series", "documents", "error"]
|
| 72 |
value: Any | None = None
|
| 73 |
columns: list[str] | None = None
|
| 74 |
rows: list[list[Any]] | None = None
|
|
|
|
| 68 |
|
| 69 |
class ToolOutput(BaseModel):
|
| 70 |
tool: str
|
| 71 |
+
kind: Literal["scalar", "table", "stats", "series", "documents", "chart", "error"]
|
| 72 |
value: Any | None = None
|
| 73 |
columns: list[str] | None = None
|
| 74 |
rows: list[list[Any]] | None = None
|
src/tools/invoker.py
CHANGED
|
@@ -36,6 +36,7 @@ from src.tools.analytics import (
|
|
| 36 |
relationship,
|
| 37 |
segmentation,
|
| 38 |
temporal,
|
|
|
|
| 39 |
)
|
| 40 |
from src.tools.contracts import ToolOutput
|
| 41 |
from src.tools.data_access import DATA_ACCESS_TOOLS, DataAccessToolInvoker
|
|
@@ -54,6 +55,7 @@ _DISPATCH: dict[str, tuple[Callable[..., Any], str]] = {
|
|
| 54 |
"analyze_segment": (segmentation.analyze_segment, "table"),
|
| 55 |
"analyze_trend": (temporal.analyze_trend, "series"),
|
| 56 |
"analyze_merge": (merge.analyze_merge, "table"),
|
|
|
|
| 57 |
}
|
| 58 |
|
| 59 |
|
|
|
|
| 36 |
relationship,
|
| 37 |
segmentation,
|
| 38 |
temporal,
|
| 39 |
+
visualization,
|
| 40 |
)
|
| 41 |
from src.tools.contracts import ToolOutput
|
| 42 |
from src.tools.data_access import DATA_ACCESS_TOOLS, DataAccessToolInvoker
|
|
|
|
| 55 |
"analyze_segment": (segmentation.analyze_segment, "table"),
|
| 56 |
"analyze_trend": (temporal.analyze_trend, "series"),
|
| 57 |
"analyze_merge": (merge.analyze_merge, "table"),
|
| 58 |
+
"render_chart": (visualization.render_chart, "chart"),
|
| 59 |
}
|
| 60 |
|
| 61 |
|
src/tools/registry.py
CHANGED
|
@@ -34,10 +34,12 @@ from src.tools.analytics import (
|
|
| 34 |
relationship,
|
| 35 |
segmentation,
|
| 36 |
temporal,
|
|
|
|
| 37 |
)
|
| 38 |
from src.tools.contracts import ToolRegistry, ToolSpec
|
| 39 |
|
| 40 |
-
# Active this round — the
|
|
|
|
| 41 |
ACTIVE_ANALYTICS_TOOLS: list[ToolSpec] = [
|
| 42 |
ToolSpec(
|
| 43 |
name="analyze_descriptive",
|
|
@@ -113,6 +115,23 @@ ACTIVE_ANALYTICS_TOOLS: list[ToolSpec] = [
|
|
| 113 |
output_kind="table",
|
| 114 |
description=merge.DESCRIPTION,
|
| 115 |
),
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
]
|
| 117 |
|
| 118 |
# Deferred this round — specs kept intact for easy re-activation, NOT exposed to
|
|
|
|
| 34 |
relationship,
|
| 35 |
segmentation,
|
| 36 |
temporal,
|
| 37 |
+
visualization,
|
| 38 |
)
|
| 39 |
from src.tools.contracts import ToolRegistry, ToolSpec
|
| 40 |
|
| 41 |
+
# Active this round — the analytics (+ render_chart, SPINE_V2_PLAN §4.1) tools
|
| 42 |
+
# the Planner may select.
|
| 43 |
ACTIVE_ANALYTICS_TOOLS: list[ToolSpec] = [
|
| 44 |
ToolSpec(
|
| 45 |
name="analyze_descriptive",
|
|
|
|
| 115 |
output_kind="table",
|
| 116 |
description=merge.DESCRIPTION,
|
| 117 |
),
|
| 118 |
+
ToolSpec(
|
| 119 |
+
name="render_chart",
|
| 120 |
+
category="analytics.visualization",
|
| 121 |
+
input_schema={
|
| 122 |
+
"required": ["data", "chart_type", "x", "y"],
|
| 123 |
+
"properties": {
|
| 124 |
+
"data": {"type": "string"},
|
| 125 |
+
"chart_type": {"type": "string"},
|
| 126 |
+
"x": {"type": "string"},
|
| 127 |
+
"y": {"type": "string"},
|
| 128 |
+
"series": {"type": "string"},
|
| 129 |
+
"title": {"type": "string"},
|
| 130 |
+
},
|
| 131 |
+
},
|
| 132 |
+
output_kind="chart",
|
| 133 |
+
description=visualization.DESCRIPTION,
|
| 134 |
+
),
|
| 135 |
]
|
| 136 |
|
| 137 |
# Deferred this round — specs kept intact for easy re-activation, NOT exposed to
|
src/traceability/scratchpad.py
CHANGED
|
@@ -71,13 +71,40 @@ def _output_to_dict(output: Any) -> dict[str, Any]:
|
|
| 71 |
]
|
| 72 |
value = getattr(output, "value", None)
|
| 73 |
if value is not None:
|
| 74 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75 |
error = getattr(output, "error", None)
|
| 76 |
if error is not None:
|
| 77 |
result["error"] = _truncate(error)
|
| 78 |
return result
|
| 79 |
|
| 80 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 81 |
def _meta_of(output: Any) -> dict[str, Any]:
|
| 82 |
"""Best-effort read of a tool result's `meta` dict (ToolOutput or plain dict)."""
|
| 83 |
if isinstance(output, dict):
|
|
@@ -109,6 +136,11 @@ def _summarize(name: str, out_dict: dict[str, Any], meta: dict[str, Any]) -> str
|
|
| 109 |
if name.startswith("analyze_"):
|
| 110 |
pretty = name.removeprefix("analyze_").replace("_", " ")
|
| 111 |
return f"Ran {pretty} analysis on {ncol} columns" if ncol else f"Ran {pretty} analysis"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 112 |
return name.replace("_", " ")
|
| 113 |
|
| 114 |
|
|
|
|
| 71 |
]
|
| 72 |
value = getattr(output, "value", None)
|
| 73 |
if value is not None:
|
| 74 |
+
if kind == "chart" and isinstance(value, dict):
|
| 75 |
+
# The full dataeyond.chart.v1 envelope is persisted in message_charts
|
| 76 |
+
# (SPINE_V2_PLAN §4.4); traceability keeps a compact summary — the
|
| 77 |
+
# spec's x/y arrays are numeric, so _truncate's string caps would let
|
| 78 |
+
# them through whole.
|
| 79 |
+
result.update(_chart_summary(value))
|
| 80 |
+
else:
|
| 81 |
+
result["value"] = _truncate(value)
|
| 82 |
error = getattr(output, "error", None)
|
| 83 |
if error is not None:
|
| 84 |
result["error"] = _truncate(error)
|
| 85 |
return result
|
| 86 |
|
| 87 |
|
| 88 |
+
def _chart_summary(envelope: dict[str, Any]) -> dict[str, Any]:
|
| 89 |
+
"""Compact wire shape for a chart-kind output: chart_type/title/trace_count/
|
| 90 |
+
point_count (charts have no rows, so `row_count` is absent — §4.7)."""
|
| 91 |
+
plotly = envelope.get("plotly")
|
| 92 |
+
traces = plotly.get("data") if isinstance(plotly, dict) else None
|
| 93 |
+
traces = traces if isinstance(traces, list) else []
|
| 94 |
+
points = 0
|
| 95 |
+
for trace in traces:
|
| 96 |
+
if isinstance(trace, dict):
|
| 97 |
+
axis = trace.get("x") if trace.get("x") is not None else trace.get("labels")
|
| 98 |
+
if isinstance(axis, list):
|
| 99 |
+
points += len(axis)
|
| 100 |
+
return {
|
| 101 |
+
"chart_type": envelope.get("chart_type"),
|
| 102 |
+
"title": envelope.get("title"),
|
| 103 |
+
"trace_count": len(traces),
|
| 104 |
+
"point_count": points,
|
| 105 |
+
}
|
| 106 |
+
|
| 107 |
+
|
| 108 |
def _meta_of(output: Any) -> dict[str, Any]:
|
| 109 |
"""Best-effort read of a tool result's `meta` dict (ToolOutput or plain dict)."""
|
| 110 |
if isinstance(output, dict):
|
|
|
|
| 136 |
if name.startswith("analyze_"):
|
| 137 |
pretty = name.removeprefix("analyze_").replace("_", " ")
|
| 138 |
return f"Ran {pretty} analysis on {ncol} columns" if ncol else f"Ran {pretty} analysis"
|
| 139 |
+
if name == "render_chart":
|
| 140 |
+
ctype = out_dict.get("chart_type") or "chart"
|
| 141 |
+
points = out_dict.get("point_count")
|
| 142 |
+
summary = f"Built a {ctype} chart"
|
| 143 |
+
return f"{summary} ({points} data points)" if points else summary
|
| 144 |
return name.replace("_", " ")
|
| 145 |
|
| 146 |
|