Rifqi Hafizuddin Claude Fable 5 commited on
Commit
87cfcf8
·
1 Parent(s): ef0c7b1

[KM-715] charts: render_chart tool + message_charts store + GET /api/v1/charts + planner viz slice

Browse files

W2 of SPINE_V2_PLAN: deterministic Plotly-JSON spec builder (dataeyond.chart.v1,
no plotly dep), ToolOutput.kind "chart", never-throw ChartStore mirroring
traceability (rows written before done), FE fetches GET /charts on done (empty
list = valid 200; done event unchanged). Planner: recipe table + explicit-ask
rule, Examples J (viz tail) + K (viz-infeasible), validator Check 10
(render_chart.data must be table-kind). Traceability records charts compactly
(point_count, not raw arrays). Live e2e vs dedorch: ALL PASS.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

API_CONTRACT_BE_PYTHON.md CHANGED
@@ -34,6 +34,7 @@ The frontend uses this service during the analysis conversation flow:
34
  | `GET` | `/api/v1/tools/report/{analysis_id}/readiness` | Report-readiness signal for the Generate-Report button (added 2026-07-09). |
35
  | `GET` | `/api/v1/tools/report/{analysis_id}/{version}` | Retrieve one report version. |
36
  | `GET` | `/api/v1/traceability` | Retrieve provenance for one assistant answer. |
 
37
 
38
  ## Common Concepts
39
 
@@ -61,6 +62,8 @@ Common event types:
61
 
62
  The stream carries answer text only. Planning, tool call details, and full provenance are fetched from `GET /api/v1/traceability` after the stream is done.
63
 
 
 
64
  ## Chat
65
 
66
  ### `POST /api/v2/chat/stream`
@@ -643,3 +646,91 @@ Frontend rendering guidance:
643
  - Default state can be collapsed.
644
  - Show planning, tool calls, and sources as separate sections.
645
  - Treat `planning: null`, `tool_calls: []`, and `sources: []` as valid states.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
34
  | `GET` | `/api/v1/tools/report/{analysis_id}/readiness` | Report-readiness signal for the Generate-Report button (added 2026-07-09). |
35
  | `GET` | `/api/v1/tools/report/{analysis_id}/{version}` | Retrieve one report version. |
36
  | `GET` | `/api/v1/traceability` | Retrieve provenance for one assistant answer. |
37
+ | `GET` | `/api/v1/charts` | Retrieve chart(s) produced by `render_chart` for one assistant answer (added 2026-07-13). |
38
 
39
  ## Common Concepts
40
 
 
62
 
63
  The stream carries answer text only. Planning, tool call details, and full provenance are fetched from `GET /api/v1/traceability` after the stream is done.
64
 
65
+ **Charts (added 2026-07-13, SPINE_V2_PLAN §4.5):** the `done` event is unchanged by charts — no additive `chart_count`/`chart_ids` field yet (open question owned by Harry). Until that's resolved, the frontend fetches `GET /api/v1/charts` unconditionally on every `done`, the same fetch-on-`done` pattern already used for traceability. SSE order and every existing field stay exactly as documented above; `sources` stays `[]`.
66
+
67
  ## Chat
68
 
69
  ### `POST /api/v2/chat/stream`
 
646
  - Default state can be collapsed.
647
  - Show planning, tool calls, and sources as separate sections.
648
  - Treat `planning: null`, `tool_calls: []`, and `sources: []` as valid states.
649
+
650
+ ## Charts
651
+
652
+ > Added 2026-07-13 (SPINE_V2_PLAN §4.4/§4.5, S2 visualization). A chart is a `render_chart` tool output, planner-selected only when the user explicitly asks to plot/visualize. Delivery mirrors traceability: a Python-owned store plus a dedicated `GET` endpoint; the streamed answer (SSE) stays text-only — charts are never embedded in `chunk`.
653
+
654
+ ### `GET /api/v1/charts`
655
+
656
+ Returns every chart produced by `render_chart` during one assistant answer.
657
+
658
+ The frontend should call this after the chat stream emits `done`, using the `analysis_id` + `message_id` from the `done` event — the same fetch-on-`done` pattern as traceability; the row(s) are written **before** `done`, so there is no polling race. Unlike `GET /api/v1/traceability`, **an empty list is a valid `200`** — most turns don't ask for a chart, and once `done` has fired there is no "not ready yet" state to distinguish from "no charts this turn", so this endpoint never 404s on a chartless turn.
659
+
660
+ Query params:
661
+
662
+ | Query | Required | Description |
663
+ | --- | --- | --- |
664
+ | `analysis_id` | Yes | Analysis identifier. |
665
+ | `message_id` | Yes | Assistant answer identifier returned by the stream. |
666
+
667
+ Example:
668
+
669
+ ```text
670
+ GET /api/v1/charts?analysis_id=an_42&message_id=msg_88f1
671
+ ```
672
+
673
+ The `dataeyond.chart.v1` envelope (the shape of `charts[].spec`, verbatim from `render_chart`):
674
+
675
+ ```json
676
+ {
677
+ "schema": "dataeyond.chart.v1",
678
+ "chart_type": "bar",
679
+ "title": "Revenue by region",
680
+ "plotly": {
681
+ "data": [{ "type": "bar", "x": ["A", "B"], "y": [1, 2], "name": "revenue" }],
682
+ "layout": { "title": {"text": "Revenue by region"}, "xaxis": {"title": {"text": "region"}},
683
+ "yaxis": {"title": {"text": "revenue"}} }
684
+ }
685
+ }
686
+ ```
687
+
688
+ Frontend renders it with `Plotly.newPlot(el, spec.plotly.data, spec.plotly.layout)`. v1 chart types: `bar`, `line`, `pie`, `scatter`.
689
+
690
+ Response `200` (one chart):
691
+
692
+ ```json
693
+ {
694
+ "count": 1,
695
+ "charts": [
696
+ {
697
+ "chart_id": "3fbd8e2e-8e21-4d4b-9b21-9e6b6a0a6a6e",
698
+ "chart_type": "bar",
699
+ "title": "Revenue by region",
700
+ "spec": {
701
+ "schema": "dataeyond.chart.v1",
702
+ "chart_type": "bar",
703
+ "title": "Revenue by region",
704
+ "plotly": {
705
+ "data": [{ "type": "bar", "x": ["A", "B"], "y": [1, 2], "name": "revenue" }],
706
+ "layout": { "title": {"text": "Revenue by region"}, "xaxis": {"title": {"text": "region"}},
707
+ "yaxis": {"title": {"text": "revenue"}} }
708
+ }
709
+ },
710
+ "created_at": "2026-07-13T03:21:09.114Z"
711
+ }
712
+ ]
713
+ }
714
+ ```
715
+
716
+ Response `200` (chartless turn — the common case; not an error):
717
+
718
+ ```json
719
+ {
720
+ "count": 0,
721
+ "charts": []
722
+ }
723
+ ```
724
+
725
+ Field rules:
726
+
727
+ - `spec` is the full `dataeyond.chart.v1` envelope, unmodified — it is the source of truth, not a projection; render straight from it.
728
+ - `chart_type` / `title` are copied out of `spec` for convenience (list rendering without parsing `spec`); `title` may be `null`.
729
+ - A turn can produce more than one chart (multiple `render_chart` calls in the same plan); `charts` is ordered by creation time.
730
+ - The payload carries no `user_id` (unlike traceability) — charts are keyed by `analysis_id` + `message_id` only.
731
+
732
+ Frontend rendering guidance:
733
+
734
+ - Fetch unconditionally on every `done` — see the SSE note above (no `chart_count` hint yet); an empty `charts[]` means render nothing extra.
735
+ - Render each chart under the assistant message it belongs to, via `Plotly.newPlot`.
736
+ - "Chart iteration" v1 = a follow-up chat turn (e.g. "make it a line chart") — the planner re-emits `render_chart` with patched args. There is no separate edit endpoint.
main.py CHANGED
@@ -22,6 +22,7 @@ from src.api.v1.report import router as report_router
22
  from src.api.v1.tools import router as tools_router
23
  from src.api.v1.help import router as help_router # pr/5 Phase 2: dedicated /tools/help
24
  from src.api.v1.traceability import router as traceability_router # KM-691
 
25
  from src.api.v2.chat import router as chat_v2_router # pr/5 Phase 2: v2 chat pilot (analysis_id)
26
  from src.db.postgres.init_db import init_db
27
  from src.config.settings import settings
@@ -68,6 +69,7 @@ app.include_router(report_router)
68
  app.include_router(tools_router)
69
  app.include_router(help_router)
70
  app.include_router(traceability_router) # KM-691: GET /api/v1/traceability
 
71
  app.include_router(chat_v2_router) # pr/5 Phase 2: POST /api/v2/chat/stream (analysis_id)
72
 
73
 
 
22
  from src.api.v1.tools import router as tools_router
23
  from src.api.v1.help import router as help_router # pr/5 Phase 2: dedicated /tools/help
24
  from src.api.v1.traceability import router as traceability_router # KM-691
25
+ from src.api.v1.charts import router as charts_router # W2: GET /api/v1/charts (SPINE_V2_PLAN §4.5)
26
  from src.api.v2.chat import router as chat_v2_router # pr/5 Phase 2: v2 chat pilot (analysis_id)
27
  from src.db.postgres.init_db import init_db
28
  from src.config.settings import settings
 
69
  app.include_router(tools_router)
70
  app.include_router(help_router)
71
  app.include_router(traceability_router) # KM-691: GET /api/v1/traceability
72
+ app.include_router(charts_router) # W2: GET /api/v1/charts (SPINE_V2_PLAN §4.5)
73
  app.include_router(chat_v2_router) # pr/5 Phase 2: POST /api/v2/chat/stream (analysis_id)
74
 
75
 
src/agents/chat_handler.py CHANGED
@@ -54,6 +54,7 @@ from .refusals import blocked_message, out_of_scope_message
54
 
55
  if TYPE_CHECKING:
56
  from ..catalog.reader import CatalogReader
 
57
  from ..retrieval.router import RetrievalRouter
58
  from ..traceability.store import TraceabilityStore
59
  from .gate import AnalysisState
@@ -105,6 +106,7 @@ class ChatHandler:
105
  ) = None,
106
  analysis_store: ReportInputStore | None = None,
107
  traceability_store: TraceabilityStore | None = None,
 
108
  check_invoker_factory: Callable[[str], Any] | None = None,
109
  ps_agent: ProblemStatementAgent | None = None,
110
  help_agent: HelpAgent | None = None,
@@ -128,6 +130,9 @@ class ChatHandler:
128
  # Traceability (KM-691): user-facing per-turn provenance store. Injectable for
129
  # tests; lazily built (Postgres) in production. Distinct from Langfuse tracing.
130
  self._traceability_store = traceability_store
 
 
 
131
  # `check` skill: builds the data-access invoker (check_data/check_knowledge)
132
  # per request with the authenticated user_id. Injectable for tests.
133
  self._check_invoker_factory = check_invoker_factory
@@ -644,6 +649,13 @@ class ChatHandler:
644
  self._traceability_store = PostgresTraceabilityStore()
645
  return self._traceability_store
646
 
 
 
 
 
 
 
 
647
  async def _flush_trace(
648
  self, pad: TraceabilityScratchpad, analysis_id: str | None, user_id: str
649
  ) -> None:
@@ -763,6 +775,22 @@ class ChatHandler:
763
  pad.set_planning_from_record(record)
764
  except Exception as e: # persistence must never break the user's answer
765
  logger.error("analysis_record persist failed", user_id=user_id, error=str(e))
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
766
  tracer.end() # output omitted (chat_answer may contain PII on Cloud)
767
  if pad is not None:
768
  await self._flush_trace(pad, analysis_id, user_id)
 
54
 
55
  if TYPE_CHECKING:
56
  from ..catalog.reader import CatalogReader
57
+ from ..charts.store import ChartStore
58
  from ..retrieval.router import RetrievalRouter
59
  from ..traceability.store import TraceabilityStore
60
  from .gate import AnalysisState
 
106
  ) = None,
107
  analysis_store: ReportInputStore | None = None,
108
  traceability_store: TraceabilityStore | None = None,
109
+ chart_store: ChartStore | None = None,
110
  check_invoker_factory: Callable[[str], Any] | None = None,
111
  ps_agent: ProblemStatementAgent | None = None,
112
  help_agent: HelpAgent | None = None,
 
130
  # Traceability (KM-691): user-facing per-turn provenance store. Injectable for
131
  # tests; lazily built (Postgres) in production. Distinct from Langfuse tracing.
132
  self._traceability_store = traceability_store
133
+ # Charts (S2, SPINE_V2_PLAN §4.4): `render_chart` output store. Injectable for
134
+ # tests; lazily built (Postgres) in production.
135
+ self._chart_store = chart_store
136
  # `check` skill: builds the data-access invoker (check_data/check_knowledge)
137
  # per request with the authenticated user_id. Injectable for tests.
138
  self._check_invoker_factory = check_invoker_factory
 
649
  self._traceability_store = PostgresTraceabilityStore()
650
  return self._traceability_store
651
 
652
+ def _get_chart_store(self) -> ChartStore:
653
+ if self._chart_store is None:
654
+ from ..charts import PostgresChartStore
655
+
656
+ self._chart_store = PostgresChartStore()
657
+ return self._chart_store
658
+
659
  async def _flush_trace(
660
  self, pad: TraceabilityScratchpad, analysis_id: str | None, user_id: str
661
  ) -> None:
 
775
  pad.set_planning_from_record(record)
776
  except Exception as e: # persistence must never break the user's answer
777
  logger.error("analysis_record persist failed", user_id=user_id, error=str(e))
778
+ # SPINE_V2_PLAN §4.4: chart rows are written before `done`; the FE fetches
779
+ # GET /api/v1/charts unconditionally on every `done` (no polling race).
780
+ try:
781
+ if pad is not None and pad.message_id:
782
+ for task_result in result.analysis_record.results_snapshot.values():
783
+ for output in task_result.outputs:
784
+ if output.kind == "chart" and isinstance(output.value, dict):
785
+ await self._get_chart_store().save(
786
+ message_id=pad.message_id,
787
+ analysis_id=analysis_id or "",
788
+ user_id=user_id,
789
+ record_id=result.analysis_record.record_id,
790
+ envelope=output.value,
791
+ )
792
+ except Exception as e: # chart persist must never break the user's answer
793
+ logger.error("chart persist failed", user_id=user_id, error=str(e))
794
  tracer.end() # output omitted (chat_answer may contain PII on Cloud)
795
  if pad is not None:
796
  await self._flush_trace(pad, analysis_id, user_id)
src/agents/planner/examples.py CHANGED
@@ -730,6 +730,119 @@ _EXAMPLE_I = TaskList(
730
  )
731
 
732
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
733
  EXAMPLES: list[tuple[str, TaskList]] = [
734
  ("Which product categories drove last quarter's revenue?", _EXAMPLE_A),
735
  ("How has monthly revenue trended by region this year, and what's unusual?", _EXAMPLE_B),
@@ -744,6 +857,8 @@ EXAMPLES: list[tuple[str, TaskList]] = [
744
  "order quantity?",
745
  _EXAMPLE_I,
746
  ),
 
 
747
  ]
748
 
749
 
 
730
  )
731
 
732
 
733
+ # --------------------------------------------------------------------------- #
734
+ # Example J — visualization tail (SPINE_V2_PLAN §4.3, recipe "viz tail").
735
+ # "Show me a bar chart of total revenue per product category."
736
+ # Shows: render_chart is planned ONLY on an explicit ask ("bar chart"), and is
737
+ # ALWAYS a tail step — its `data` consumes a TABLE-producing upstream (here the
738
+ # grouped top-N-style retrieve from Example G, without the limit), never stats/
739
+ # series/metadata. `x`/`y` reference that table's column aliases; the chart
740
+ # carries the already-aggregated rows (one bar per category), never raw order
741
+ # rows. The `assumptions` line carries the feasibility check on purpose: a chart
742
+ # ask never licenses aliasing a stand-in column (see Example K).
743
+ # --------------------------------------------------------------------------- #
744
+
745
+ _EXAMPLE_J = TaskList(
746
+ plan_id="example_j",
747
+ goal_restated="Chart total revenue per product category as a bar chart.",
748
+ assumptions=[
749
+ "The chart dimension exists in the catalog (c_category) — a chart ask is "
750
+ "planned only when the asked-for dimension and measure have real catalog "
751
+ "columns; otherwise it is infeasible, chart or no chart."
752
+ ],
753
+ open_questions=[],
754
+ tasks=[
755
+ Task(
756
+ id="t1",
757
+ stage="data_understanding",
758
+ objective="Confirm the sales source exposes category and revenue.",
759
+ tool_calls=[ToolCall(tool="check_data", args={"source_id": "src_sales"})],
760
+ expected_output="source_shape",
761
+ success_criteria="Produced the orders table schema; category and revenue present.",
762
+ depends_on=[],
763
+ estimated_cost="low",
764
+ ),
765
+ Task(
766
+ id="t2",
767
+ stage="data_preparation",
768
+ objective="Aggregate total revenue per product category.",
769
+ tool_calls=[
770
+ ToolCall(
771
+ tool="retrieve_data",
772
+ args={
773
+ "ir": {
774
+ "source_id": "src_sales",
775
+ "table_id": "t_orders",
776
+ "select": [
777
+ {"kind": "column", "column_id": "c_category", "alias": "category"},
778
+ {
779
+ "kind": "agg",
780
+ "fn": "sum",
781
+ "column_id": "c_revenue",
782
+ "alias": "total_revenue",
783
+ },
784
+ ],
785
+ "group_by": ["c_category"],
786
+ }
787
+ },
788
+ )
789
+ ],
790
+ expected_output="revenue_per_category",
791
+ success_criteria="Produced one total-revenue row per category.",
792
+ depends_on=["t1"],
793
+ estimated_cost="low",
794
+ ),
795
+ Task(
796
+ id="t3",
797
+ stage="evaluation",
798
+ objective="Render the per-category revenue table as a bar chart.",
799
+ tool_calls=[
800
+ ToolCall(
801
+ tool="render_chart",
802
+ args={
803
+ # `data` references the TABLE task (t2) — a chart is always a
804
+ # tail on an aggregated table, never on raw rows or stats.
805
+ "data": "${t2}",
806
+ "chart_type": "bar",
807
+ "x": "category",
808
+ "y": "total_revenue",
809
+ "title": "Total revenue by product category",
810
+ },
811
+ )
812
+ ],
813
+ expected_output="revenue_bar_chart",
814
+ success_criteria="Produced a bar-chart spec with one bar per category.",
815
+ depends_on=["t2"],
816
+ estimated_cost="low",
817
+ ),
818
+ ],
819
+ )
820
+
821
+
822
+ # --------------------------------------------------------------------------- #
823
+ # Example K — a chart ask that is INFEASIBLE (SPINE_V2_PLAN §4.3 + planner.md
824
+ # "Charts ... never relaxes feasibility"). "Plot the customer churn rate by
825
+ # month as a line chart" against the sales catalog: churn does not exist, and
826
+ # the explicit chart request does NOT license mapping some other column into
827
+ # the asked-for measure — the verdict is the same infeasible_reason Example H
828
+ # would give, chart or no chart.
829
+ # --------------------------------------------------------------------------- #
830
+
831
+ _EXAMPLE_K = TaskList(
832
+ plan_id="example_k",
833
+ goal_restated="Chart the monthly customer churn rate as a line chart.",
834
+ assumptions=[],
835
+ open_questions=[],
836
+ tasks=[],
837
+ infeasible_reason=(
838
+ "The connected source has no churn or subscription-status data — the "
839
+ "orders table only carries order-level category, revenue, quantity, and "
840
+ "dates — so there is nothing to chart. Nearest chartable analyses: "
841
+ "monthly revenue trend, or order counts per category."
842
+ ),
843
+ )
844
+
845
+
846
  EXAMPLES: list[tuple[str, TaskList]] = [
847
  ("Which product categories drove last quarter's revenue?", _EXAMPLE_A),
848
  ("How has monthly revenue trended by region this year, and what's unusual?", _EXAMPLE_B),
 
857
  "order quantity?",
858
  _EXAMPLE_I,
859
  ),
860
+ ("Show me a bar chart of total revenue per product category.", _EXAMPLE_J),
861
+ ("Plot the customer churn rate by month as a line chart.", _EXAMPLE_K),
862
  ]
863
 
864
 
src/agents/planner/validator.py CHANGED
@@ -281,6 +281,24 @@ class PlannerValidator:
281
  "analyzable data rows. Feed analyze_* from a data-producing tool "
282
  "(e.g. retrieve_data)."
283
  )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
284
 
285
  @staticmethod
286
  def _validate_dag(tasks_by_id: dict, id_set: set[str]) -> None:
 
281
  "analyzable data rows. Feed analyze_* from a data-producing tool "
282
  "(e.g. retrieve_data)."
283
  )
284
+ # Check 10 — render_chart path shape (SPINE_V2_PLAN §4.3): the chart's
285
+ # `data` must come from a task whose tool yields a TABLE. A stats- or
286
+ # series-kind upstream passes the category denylist above but cannot
287
+ # be materialized into a DataFrame at execution, so the chart could
288
+ # only fail late; reject pre-run so the retry re-prompts toward a
289
+ # grouped retrieve_data or a table-producing analyze_*.
290
+ if (
291
+ call.tool == "render_chart"
292
+ and ref_spec is not None
293
+ and ref_spec.output_kind != "table"
294
+ ):
295
+ raise PlannerValidationError(
296
+ f"task {task_id}: render_chart takes its {arg_name!r} from task "
297
+ f"{match.group(1)} ({ref_tool!r}), which yields "
298
+ f"{ref_spec.output_kind!r} output — a chart is drawn from a TABLE. "
299
+ "Feed it a table-producing task (a grouped retrieve_data, "
300
+ "analyze_aggregate, analyze_merge, ...)."
301
+ )
302
 
303
  @staticmethod
304
  def _validate_dag(tasks_by_id: dict, id_set: set[str]) -> None:
src/api/v1/charts.py ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Charts endpoint — user-facing chart retrieval for one turn (S2 visualization,
2
+ SPINE_V2_PLAN §4.4/§4.5).
3
+
4
+ `GET /api/v1/charts?analysis_id=&message_id=` returns every chart a `render_chart`
5
+ tool call produced during one assistant turn, as `dataeyond.chart.v1` envelopes
6
+ (SPINE_V2_PLAN §4.2). Rows are written by the chat pipeline right before the `done`
7
+ SSE event; the FE fires this GET on `done`.
8
+
9
+ Unlike `GET /api/v1/traceability`, an **empty list is a valid 200** here (most turns
10
+ don't ask for a chart) rather than a 404 — once `done` has fired there is no "not
11
+ ready yet" state to distinguish from "no charts this turn".
12
+
13
+ No auth — Go fronts Python.
14
+ """
15
+
16
+ from fastapi import APIRouter, Query
17
+ from pydantic import BaseModel
18
+
19
+ from src.charts import ChartRecord, PostgresChartStore
20
+ from src.middlewares.logging import get_logger, log_execution
21
+
22
+ logger = get_logger("charts_api")
23
+
24
+ router = APIRouter(prefix="/api/v1", tags=["Charts"])
25
+
26
+ # Warm, process-shared store (mirrors the chat endpoints' module-level instances).
27
+ _store = PostgresChartStore()
28
+
29
+
30
+ class ChartsResponse(BaseModel):
31
+ count: int
32
+ charts: list[ChartRecord]
33
+
34
+
35
+ @router.get("/charts", response_model=ChartsResponse)
36
+ @log_execution(logger)
37
+ async def get_charts(
38
+ analysis_id: str = Query(..., description="Analysis/session id"),
39
+ message_id: str = Query(
40
+ ..., description="Assistant turn id, taken from the `done` SSE event"
41
+ ),
42
+ ) -> ChartsResponse:
43
+ """Fetch every chart for one turn. Empty list (not 404) when the turn produced
44
+ none — the FE can call this unconditionally on every `done`."""
45
+ charts = await _store.list_for_message(analysis_id, message_id)
46
+ return ChartsResponse(count=len(charts), charts=charts)
src/charts/__init__.py ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Charts (S2 visualization, SPINE_V2_PLAN §4.4): `render_chart` output persistence.
2
+
3
+ Chart envelopes produced by the `render_chart` tool (`dataeyond.chart.v1`, SPINE_V2_PLAN
4
+ §4.2) are persisted through a `ChartStore` right before the `done` SSE event, and served
5
+ by `GET /api/v1/charts`. SSE stays text-only — charts are never embedded in the streamed
6
+ answer.
7
+ """
8
+
9
+ from .store import (
10
+ ChartRecord,
11
+ ChartStore,
12
+ NullChartStore,
13
+ PostgresChartStore,
14
+ )
15
+
16
+ __all__ = [
17
+ "ChartRecord",
18
+ "ChartStore",
19
+ "NullChartStore",
20
+ "PostgresChartStore",
21
+ ]
src/charts/store.py ADDED
@@ -0,0 +1,154 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """ChartStore — the seam the chat pipeline persists `render_chart` outputs through
2
+ (S2 visualization, SPINE_V2_PLAN §4.4).
3
+
4
+ `ChatHandler._run_slow_path` scans a completed `AnalysisRecord.results_snapshot` for
5
+ `ToolOutput(kind="chart")` entries and flushes one `MessageChartRow` per chart through
6
+ this seam, right before the `done` SSE event (mirrors the traceability flush). Unlike
7
+ traceability's one row per turn, a turn with multiple `render_chart` calls writes
8
+ multiple rows sharing one `message_id`. `GET /api/v1/charts` reads them back by
9
+ (analysis_id, message_id).
10
+
11
+ - `NullChartStore` logs the envelope and stores nothing (tests / disabled persistence).
12
+ - `PostgresChartStore` writes one `message_charts` row per chart (dedorch,
13
+ `AsyncSessionLocal`), mirroring `PostgresTraceabilityStore`.
14
+
15
+ `save` must NEVER raise on the caller's path — a chart-persist failure must not break
16
+ the user's answer. `list_for_message` is the endpoint read; an empty list is a valid
17
+ result (chartless turn), not an error.
18
+ """
19
+
20
+ from __future__ import annotations
21
+
22
+ import uuid
23
+ from datetime import datetime
24
+ from typing import Protocol, runtime_checkable
25
+
26
+ from pydantic import BaseModel
27
+ from sqlalchemy import select
28
+
29
+ from src.db.postgres.connection import AsyncSessionLocal
30
+ from src.db.postgres.models import MessageChartRow
31
+ from src.middlewares.logging import get_logger
32
+
33
+ logger = get_logger("charts_store")
34
+
35
+
36
+ class ChartRecord(BaseModel):
37
+ """One persisted chart, as served by `GET /api/v1/charts`."""
38
+
39
+ chart_id: str
40
+ chart_type: str
41
+ title: str | None = None
42
+ spec: dict
43
+ created_at: datetime
44
+
45
+
46
+ @runtime_checkable
47
+ class ChartStore(Protocol):
48
+ """Persist + read `render_chart` outputs for one assistant `message_id`.
49
+
50
+ `save` must never raise on the caller's path. `list_for_message` returns every
51
+ chart for one turn (possibly empty — a turn need not have asked for a chart).
52
+ """
53
+
54
+ async def save(
55
+ self,
56
+ *,
57
+ message_id: str,
58
+ analysis_id: str,
59
+ user_id: str,
60
+ record_id: str | None,
61
+ envelope: dict,
62
+ ) -> None: ...
63
+
64
+ async def list_for_message(
65
+ self, analysis_id: str, message_id: str
66
+ ) -> list[ChartRecord]: ...
67
+
68
+
69
+ class NullChartStore:
70
+ """No-op store: logs the envelope, persists nothing. Reads return `[]`."""
71
+
72
+ async def save(
73
+ self,
74
+ *,
75
+ message_id: str,
76
+ analysis_id: str,
77
+ user_id: str,
78
+ record_id: str | None,
79
+ envelope: dict,
80
+ ) -> None:
81
+ logger.info(
82
+ "chart produced (not persisted — NullChartStore)",
83
+ message_id=message_id,
84
+ chart_type=envelope.get("chart_type", "unknown"),
85
+ )
86
+
87
+ async def list_for_message(
88
+ self, analysis_id: str, message_id: str
89
+ ) -> list[ChartRecord]:
90
+ return []
91
+
92
+
93
+ class PostgresChartStore:
94
+ """Writes/reads `message_charts` rows. One insert per chart (not an upsert)."""
95
+
96
+ async def save(
97
+ self,
98
+ *,
99
+ message_id: str,
100
+ analysis_id: str,
101
+ user_id: str,
102
+ record_id: str | None,
103
+ envelope: dict,
104
+ ) -> None:
105
+ try:
106
+ async with AsyncSessionLocal() as session:
107
+ row = MessageChartRow(
108
+ id=str(uuid.uuid4()),
109
+ message_id=message_id,
110
+ analysis_id=analysis_id,
111
+ user_id=user_id,
112
+ record_id=record_id,
113
+ chart_type=envelope.get("chart_type", "unknown"),
114
+ title=envelope.get("title"),
115
+ spec=envelope,
116
+ )
117
+ session.add(row)
118
+ await session.commit()
119
+ logger.info(
120
+ "chart persisted",
121
+ message_id=message_id,
122
+ analysis_id=analysis_id,
123
+ chart_type=envelope.get("chart_type", "unknown"),
124
+ )
125
+ except Exception as exc: # never break the user's answer
126
+ logger.error(
127
+ "chart persist failed",
128
+ message_id=message_id,
129
+ error=str(exc),
130
+ )
131
+
132
+ async def list_for_message(
133
+ self, analysis_id: str, message_id: str
134
+ ) -> list[ChartRecord]:
135
+ async with AsyncSessionLocal() as session:
136
+ result = await session.execute(
137
+ select(MessageChartRow)
138
+ .where(
139
+ MessageChartRow.analysis_id == analysis_id,
140
+ MessageChartRow.message_id == message_id,
141
+ )
142
+ .order_by(MessageChartRow.created_at)
143
+ )
144
+ rows = result.scalars().all()
145
+ return [
146
+ ChartRecord(
147
+ chart_id=row.id,
148
+ chart_type=row.chart_type,
149
+ title=row.title,
150
+ spec=row.spec,
151
+ created_at=row.created_at,
152
+ )
153
+ for row in rows
154
+ ]
src/config/prompts/planner.md CHANGED
@@ -20,7 +20,9 @@ only a `TaskList` object that conforms to the provided schema.
20
  section. Copy the stable ids verbatim — downstream validation does a literal
21
  id lookup, so a paraphrased name fails.
22
  5. **No modeling in v1.** There are no modeling tools. Do not emit `modeling`
23
- tasks. The product is descriptive/diagnostic only — no predictions, no charts.
 
 
24
  6. **Never re-purpose a column as a different business measure.** A column means
25
  what the catalog says it means — do not alias one concept as another to force
26
  an answer (e.g. selecting an availability percentage AS "revenue", or a
@@ -35,6 +37,21 @@ only a `TaskList` object that conforms to the provided schema.
35
  is noise, not helpfulness. A multi-part task list is correct ONLY when the
36
  question itself has multiple parts (e.g. "trend by region AND what's unusual").
37
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
38
  # How to plan
39
 
40
  - **Smallest plan that answers the question.** Do not exceed `Constraints.max_tasks`.
@@ -82,6 +99,20 @@ only a `TaskList` object that conforms to the provided schema.
82
  to a task's LAST output, so two retrievals inside one task lose the first
83
  table. The merged table (one row per entity, both measures) answers the
84
  question, or feeds a further `analyze_*` step.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
85
  - **Mixing structured + unstructured.** If qualitative context helps, add a
86
  `retrieve_knowledge` task against an unstructured source listed in the catalog.
87
  - **CRISP-DM stages.** Tag each task with the stage it serves:
 
20
  section. Copy the stable ids verbatim — downstream validation does a literal
21
  id lookup, so a paraphrased name fails.
22
  5. **No modeling in v1.** There are no modeling tools. Do not emit `modeling`
23
+ tasks. The product is descriptive/diagnostic only — no predictions. A chart
24
+ exists ONLY as a `render_chart` tail on an explicit user ask (see "Charts"
25
+ under "How to plan") — never speculatively.
26
  6. **Never re-purpose a column as a different business measure.** A column means
27
  what the catalog says it means — do not alias one concept as another to force
28
  an answer (e.g. selecting an availability percentage AS "revenue", or a
 
37
  is noise, not helpfulness. A multi-part task list is correct ONLY when the
38
  question itself has multiple parts (e.g. "trend by region AND what's unusual").
39
 
40
+ # Recipes — the named workflows
41
+
42
+ Classify the question into ONE recipe, then instantiate it. Most questions are a
43
+ recipe verbatim; a genuinely multi-part question composes recipes.
44
+
45
+ | Recipe | The question asks for… | Chain |
46
+ |---|---|---|
47
+ | R1 descriptive | a summary/distribution of columns | `retrieve_data` → `analyze_descriptive` |
48
+ | R2 aggregate / top-N | totals or averages per group, "top N by …" | ONE grouped `retrieve_data` IR (± `analyze_aggregate`) |
49
+ | R3 trend | movement over time | `retrieve_data` → `analyze_trend` |
50
+ | R4 correlation | the relationship between numeric columns | `retrieve_data` → `analyze_correlation` |
51
+ | R5 two-metric merge | "which X has both A and B" | `retrieve_data` ×2 → `analyze_merge` → … |
52
+ | R6 infeasible | a measure/entity no catalog column holds | `tasks: []` + `infeasible_reason` |
53
+ | viz tail | …any of the above, WITH an explicit ask to chart it | R1–R5 + a `render_chart` tail |
54
+
55
  # How to plan
56
 
57
  - **Smallest plan that answers the question.** Do not exceed `Constraints.max_tasks`.
 
99
  to a task's LAST output, so two retrievals inside one task lose the first
100
  table. The merged table (one row per entity, both measures) answers the
101
  question, or feeds a further `analyze_*` step.
102
+ - **Charts (`render_chart`) only on an explicit ask.** Add a `render_chart` tail
103
+ ONLY when the user explicitly asks to see a chart — "plot", "chart", "graph",
104
+ "visualize", "diagram" (ID: "grafik", "diagram", "visualisasikan", "buatkan
105
+ grafik/diagram") — NEVER speculatively on a plain data question. It is always
106
+ the LAST step: its `data` takes a `"${t<id>}"` from a task that yields a
107
+ **table** (a grouped `retrieve_data` or a table-producing `analyze_*` — never
108
+ stats/series/metadata), and `x`/`y` are that table's column aliases. Chart the
109
+ already-aggregated table (one row per category/period), not raw rows. Pick
110
+ `chart_type` by the question: `bar` (magnitude per category), `line` (over
111
+ time), `pie` (share of a small whole), `scatter` (two numeric columns).
112
+ A chart ask NEVER relaxes feasibility (rule 6): if the asked-for dimension or
113
+ measure has no catalog column, the question is **infeasible** — never chart a
114
+ stand-in column aliased under the asked-for name (e.g. never select a status
115
+ column AS "region" because the user asked for a chart by region).
116
  - **Mixing structured + unstructured.** If qualitative context helps, add a
117
  `retrieve_knowledge` task against an unstructured source listed in the catalog.
118
  - **CRISP-DM stages.** Tag each task with the stage it serves:
src/db/postgres/models.py CHANGED
@@ -284,3 +284,36 @@ class MessageTraceabilityRow(Base):
284
  intent = Column(String, nullable=False)
285
  data = Column(JSONB, nullable=False) # full TraceabilityPayload (source of truth)
286
  created_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
284
  intent = Column(String, nullable=False)
285
  data = Column(JSONB, nullable=False) # full TraceabilityPayload (source of truth)
286
  created_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
287
+
288
+
289
+ class MessageChartRow(Base):
290
+ """One row per chart produced by `render_chart` — S2 visualization (SPINE_V2_PLAN §4.4).
291
+
292
+ `spec` holds the full `dataeyond.chart.v1` envelope (SPINE_V2_PLAN §4.2) exactly as
293
+ returned by the tool — `{schema, chart_type, title, plotly: {data, layout}}` — it is
294
+ the source of truth the FE renders with
295
+ `Plotly.newPlot(el, spec.plotly.data, spec.plotly.layout)`. Unlike
296
+ `MessageTraceabilityRow` (one row per turn), **multiple rows can share one
297
+ `message_id`** — one row per chart a turn produced. Written before the `done` SSE
298
+ event and served by `GET /api/v1/charts` (SPINE_V2_PLAN §4.4/§4.5).
299
+
300
+ OWNERSHIP / HANDOFF (SPINE_V2_PLAN §4.4, 2026-07-13): **Python-owned for now**, the
301
+ same pattern as `report_inputs` / `message_traceability`. Post-cutover `init_db` no
302
+ longer runs `create_all`, so the table is created by a one-time manual DDL run
303
+ against dedorch (Rifqi, 2026-07-13); the finalized schema goes to Harry so the
304
+ dedorch migration re-creates it later (`analysis_id` gains the FK to `analyses(id)`
305
+ there).
306
+ """
307
+ __tablename__ = "message_charts"
308
+
309
+ # Client-minted default — the DDL's `gen_random_uuid()` default never fires from
310
+ # Python (we always pass `id` explicitly, but the default is here for parity).
311
+ id = Column(UUID(as_uuid=False), primary_key=True, default=lambda: str(uuid4()))
312
+ message_id = Column(String, nullable=False, index=True) # turn id; multiple rows per turn
313
+ analysis_id = Column(UUID(as_uuid=False), nullable=False, index=True) # analysis session id
314
+ user_id = Column(String, nullable=False)
315
+ record_id = Column(String, nullable=True)
316
+ chart_type = Column(String, nullable=False)
317
+ title = Column(String, nullable=True)
318
+ spec = Column(JSONB, nullable=False) # full dataeyond.chart.v1 envelope (source of truth)
319
+ created_at = Column(DateTime(timezone=True), nullable=False, server_default=func.now())
src/tools/analytics/visualization.py ADDED
@@ -0,0 +1,239 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """render_chart — declarative chart-spec builder (SPINE_V2_PLAN §4.1, S2).
2
+
3
+ First tool of the `render_*` family: turns an already-materialized DataFrame into
4
+ a Plotly-conformant JSON spec wrapped in the `dataeyond.chart.v1` envelope
5
+ (SPINE_V2_PLAN §4.2). The FE renders it with plotly.js
6
+ (`Plotly.newPlot(el, spec.plotly.data, spec.plotly.layout)`).
7
+
8
+ Deliberately NOT a code generator and NOT a plotly import (locked decision,
9
+ DEV_PLAN deferred row #26): the spec is a hand-built dict, so no generated code
10
+ ever executes and no new dependency lands. Style (colors, axis chrome) is the
11
+ fixed module preset below — never a planner/LLM decision (CoDA's "Design
12
+ Explorer" phase collapsed to a constant).
13
+
14
+ Pattern A, same as the `analyze_*` family: `data` is resolved to a DataFrame by
15
+ the invoker from an upstream table-kind output; this function never fetches.
16
+ """
17
+
18
+ from __future__ import annotations
19
+
20
+ import datetime as _dt
21
+ from decimal import Decimal
22
+
23
+ import numpy as np
24
+ import pandas as pd
25
+
26
+ from src.tools.analytics.descriptive import ColumnNotFoundError
27
+
28
+ # v1 chart types (SPINE_V2_PLAN §4.1). `pie` maps x -> labels, y -> values.
29
+ CHART_TYPES = ("bar", "line", "pie", "scatter")
30
+
31
+
32
+ class UnsupportedChartTypeError(ValueError):
33
+ """The requested chart_type is not in CHART_TYPES (error_code UNSUPPORTED_CHART_TYPE)."""
34
+
35
+
36
+ # --------------------------------------------------------------------------- #
37
+ # House style preset — fixed by design, not a planner argument.
38
+ #
39
+ # Colorway = the dataviz reference categorical palette (8 slots, light mode).
40
+ # The slot ORDER is the colorblind-safety mechanism (validated: worst adjacent
41
+ # CVD deltaE 24.2 vs the >=12 target) — series take slots in this fixed order,
42
+ # never cycled or re-picked. Backgrounds are transparent so the chart inherits
43
+ # the FE surface; inks/gridlines are the palette's chrome roles.
44
+ # --------------------------------------------------------------------------- #
45
+ COLORWAY = [
46
+ "#2a78d6", # blue
47
+ "#1baf7a", # aqua
48
+ "#eda100", # yellow
49
+ "#008300", # green
50
+ "#4a3aa7", # violet
51
+ "#e34948", # red
52
+ "#e87ba4", # magenta
53
+ "#eb6834", # orange
54
+ ]
55
+
56
+ _INK_SECONDARY = "#52514e" # body text
57
+ _INK_MUTED = "#898781" # axis titles / tick labels (mode-neutral)
58
+ _GRIDLINE = "#e1e0d9" # hairline grid
59
+ _BASELINE = "#c3c2b7" # axis line
60
+
61
+ LAYOUT_PRESET: dict[str, object] = {
62
+ "colorway": COLORWAY,
63
+ "font": {
64
+ "family": 'system-ui, -apple-system, "Segoe UI", sans-serif',
65
+ "size": 13,
66
+ "color": _INK_SECONDARY,
67
+ },
68
+ "paper_bgcolor": "rgba(0,0,0,0)",
69
+ "plot_bgcolor": "rgba(0,0,0,0)",
70
+ "margin": {"t": 48, "r": 16, "b": 48, "l": 56},
71
+ "bargap": 0.25, # keeps adjacent bars visually separated (mark-spec gap)
72
+ }
73
+
74
+ # Mark specs: thin marks — 2px lines, 8px markers.
75
+ _LINE_STYLE = {"width": 2}
76
+ _MARKER_STYLE = {"size": 8}
77
+
78
+
79
+ def _axis(label: str) -> dict[str, object]:
80
+ """Recessive axis chrome + the column name as the axis title."""
81
+ return {
82
+ "title": {"text": label, "font": {"color": _INK_MUTED}},
83
+ "tickfont": {"color": _INK_MUTED},
84
+ "gridcolor": _GRIDLINE,
85
+ "linecolor": _BASELINE,
86
+ "automargin": True,
87
+ }
88
+
89
+
90
+ def _clean(value: object) -> object:
91
+ """One JSON-safe scalar: numpy -> Python, NaN/NaT -> None, dates -> ISO strings.
92
+
93
+ Chart specs are persisted as JSONB and shipped to the FE, so every value in
94
+ the trace arrays must survive json.dumps. Upstream `_materialize` already
95
+ normalizes fully-numeric columns, but mixed columns can still carry Decimal,
96
+ and date columns carry Timestamps.
97
+ """
98
+ if value is None or value is pd.NaT:
99
+ return None
100
+ if isinstance(value, float) and np.isnan(value):
101
+ return None
102
+ if hasattr(value, "item"): # numpy scalar (incl. np.datetime64 via Timestamp below)
103
+ value = value.item()
104
+ if isinstance(value, float) and np.isnan(value):
105
+ return None
106
+ if isinstance(value, Decimal):
107
+ return float(value)
108
+ if isinstance(value, pd.Timestamp):
109
+ return value.isoformat()
110
+ if isinstance(value, _dt.datetime | _dt.date):
111
+ return value.isoformat()
112
+ if isinstance(value, bool | int | float | str):
113
+ return value
114
+ return str(value)
115
+
116
+
117
+ def _values(col: pd.Series) -> list[object]:
118
+ return [_clean(v) for v in col.tolist()]
119
+
120
+
121
+ def _series_label(value: object) -> str:
122
+ """Trace name for one series group; null groups get an explicit label."""
123
+ cleaned = _clean(value)
124
+ return "(missing)" if cleaned is None else str(cleaned)
125
+
126
+
127
+ # Prompt-style description read by the Planner to decide WHEN to pick this tool.
128
+ DESCRIPTION = """\
129
+ Summary: Build a chart specification (bar, line, pie, scatter) from an \
130
+ already-retrieved table, for the app to render. Maps one x column and one y \
131
+ column (pie: x -> slice labels, y -> slice values); an optional `series` column \
132
+ splits bar/line/scatter into one trace per category. Produces a spec only — it \
133
+ computes no numbers.
134
+
135
+ USE WHEN the user EXPLICITLY asks to see a chart — "plot", "chart", "graph", \
136
+ "visualize", "diagram" (ID: "grafik", "diagram", "plot", "visualisasikan", \
137
+ "buatkan grafik/diagram"). ALWAYS a tail step: `data` consumes the output of an \
138
+ upstream task that yields a table (retrieve_data or a table-kind analyze_*).
139
+
140
+ DON'T USE WHEN:
141
+ - the user did not explicitly ask for a chart -> answer with tables/stats only \
142
+ (never add a speculative chart)
143
+ - the numbers still need computing -> run the analyze_*/retrieve step first, \
144
+ then chart its table output
145
+ - the upstream output is stats- or series-kind -> feed it a table-kind task \
146
+ (e.g. a grouped retrieve or analyze_aggregate)
147
+
148
+ Example questions:
149
+ - "plot revenue by region as a bar chart"
150
+ - "buatkan grafik penjualan per kategori"
151
+ - "show a pie chart of orders by sales channel"
152
+ - "visualize monthly revenue" (chain a month-grouped table first, then line)
153
+ """
154
+
155
+
156
+ def render_chart(
157
+ df: pd.DataFrame,
158
+ chart_type: str,
159
+ x: str,
160
+ y: str,
161
+ series: str | None = None,
162
+ title: str | None = None,
163
+ ) -> dict[str, object]:
164
+ """Build a `dataeyond.chart.v1` envelope from a table (SPINE_V2_PLAN §4.2).
165
+
166
+ Args:
167
+ df: already-materialized data (the invoker resolves `data` upstream).
168
+ x: column for the x axis (pie: slice labels).
169
+ y: column for the y axis (pie: slice values).
170
+ series: optional column splitting bar/line/scatter into one trace per
171
+ distinct value (fixed-order colorway slots). Ignored for pie.
172
+ title: chart title; defaults to "<y> by <x>".
173
+
174
+ Returns:
175
+ The envelope dict: {schema, chart_type, title, plotly: {data, layout}}.
176
+
177
+ Raises:
178
+ UnsupportedChartTypeError: chart_type not in CHART_TYPES.
179
+ ColumnNotFoundError: x, y, or series absent from the data.
180
+ """
181
+ if chart_type not in CHART_TYPES:
182
+ raise UnsupportedChartTypeError(
183
+ f"unsupported chart_type '{chart_type}'; supported: {list(CHART_TYPES)}"
184
+ )
185
+ needed = [x, y] if chart_type == "pie" or series is None else [x, y, series]
186
+ missing = [c for c in needed if c not in df.columns]
187
+ if missing:
188
+ raise ColumnNotFoundError(f"columns not found: {missing}")
189
+
190
+ chart_title = title or f"{y} by {x}"
191
+
192
+ if chart_type == "pie":
193
+ traces: list[dict[str, object]] = [
194
+ {"type": "pie", "labels": _values(df[x]), "values": _values(df[y])}
195
+ ]
196
+ else:
197
+ plotly_type = "bar" if chart_type == "bar" else "scatter"
198
+ mode = {"line": "lines+markers", "scatter": "markers"}.get(chart_type)
199
+
200
+ def _trace(frame: pd.DataFrame, name: str) -> dict[str, object]:
201
+ t: dict[str, object] = {
202
+ "type": plotly_type,
203
+ "x": _values(frame[x]),
204
+ "y": _values(frame[y]),
205
+ "name": name,
206
+ }
207
+ if mode is not None:
208
+ t["mode"] = mode
209
+ t["line"] = dict(_LINE_STYLE)
210
+ t["marker"] = dict(_MARKER_STYLE)
211
+ return t
212
+
213
+ if series is None:
214
+ traces = [_trace(df, y)]
215
+ else:
216
+ # sort=True keeps trace order (and therefore colorway slot
217
+ # assignment) deterministic across runs of the same data.
218
+ traces = [
219
+ _trace(group, _series_label(value))
220
+ for value, group in df.groupby(series, dropna=False, sort=True)
221
+ ]
222
+
223
+ layout: dict[str, object] = {
224
+ **LAYOUT_PRESET,
225
+ "title": {"text": chart_title},
226
+ # Legend only when there is more than one thing to identify: multi-series
227
+ # traces, or pie slices (a single named series is titled, not legended).
228
+ "showlegend": len(traces) > 1 or chart_type == "pie",
229
+ }
230
+ if chart_type != "pie":
231
+ layout["xaxis"] = _axis(x)
232
+ layout["yaxis"] = _axis(y)
233
+
234
+ return {
235
+ "schema": "dataeyond.chart.v1",
236
+ "chart_type": chart_type,
237
+ "title": chart_title,
238
+ "plotly": {"data": traces, "layout": layout},
239
+ }
src/tools/contracts.py CHANGED
@@ -68,7 +68,7 @@ class ToolRegistry(BaseModel):
68
 
69
  class ToolOutput(BaseModel):
70
  tool: str
71
- kind: Literal["scalar", "table", "stats", "series", "documents", "error"]
72
  value: Any | None = None
73
  columns: list[str] | None = None
74
  rows: list[list[Any]] | None = None
 
68
 
69
  class ToolOutput(BaseModel):
70
  tool: str
71
+ kind: Literal["scalar", "table", "stats", "series", "documents", "chart", "error"]
72
  value: Any | None = None
73
  columns: list[str] | None = None
74
  rows: list[list[Any]] | None = None
src/tools/invoker.py CHANGED
@@ -36,6 +36,7 @@ from src.tools.analytics import (
36
  relationship,
37
  segmentation,
38
  temporal,
 
39
  )
40
  from src.tools.contracts import ToolOutput
41
  from src.tools.data_access import DATA_ACCESS_TOOLS, DataAccessToolInvoker
@@ -54,6 +55,7 @@ _DISPATCH: dict[str, tuple[Callable[..., Any], str]] = {
54
  "analyze_segment": (segmentation.analyze_segment, "table"),
55
  "analyze_trend": (temporal.analyze_trend, "series"),
56
  "analyze_merge": (merge.analyze_merge, "table"),
 
57
  }
58
 
59
 
 
36
  relationship,
37
  segmentation,
38
  temporal,
39
+ visualization,
40
  )
41
  from src.tools.contracts import ToolOutput
42
  from src.tools.data_access import DATA_ACCESS_TOOLS, DataAccessToolInvoker
 
55
  "analyze_segment": (segmentation.analyze_segment, "table"),
56
  "analyze_trend": (temporal.analyze_trend, "series"),
57
  "analyze_merge": (merge.analyze_merge, "table"),
58
+ "render_chart": (visualization.render_chart, "chart"),
59
  }
60
 
61
 
src/tools/registry.py CHANGED
@@ -34,10 +34,12 @@ from src.tools.analytics import (
34
  relationship,
35
  segmentation,
36
  temporal,
 
37
  )
38
  from src.tools.contracts import ToolRegistry, ToolSpec
39
 
40
- # Active this round — the four analytics tools the Planner may select.
 
41
  ACTIVE_ANALYTICS_TOOLS: list[ToolSpec] = [
42
  ToolSpec(
43
  name="analyze_descriptive",
@@ -113,6 +115,23 @@ ACTIVE_ANALYTICS_TOOLS: list[ToolSpec] = [
113
  output_kind="table",
114
  description=merge.DESCRIPTION,
115
  ),
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
116
  ]
117
 
118
  # Deferred this round — specs kept intact for easy re-activation, NOT exposed to
 
34
  relationship,
35
  segmentation,
36
  temporal,
37
+ visualization,
38
  )
39
  from src.tools.contracts import ToolRegistry, ToolSpec
40
 
41
+ # Active this round — the analytics (+ render_chart, SPINE_V2_PLAN §4.1) tools
42
+ # the Planner may select.
43
  ACTIVE_ANALYTICS_TOOLS: list[ToolSpec] = [
44
  ToolSpec(
45
  name="analyze_descriptive",
 
115
  output_kind="table",
116
  description=merge.DESCRIPTION,
117
  ),
118
+ ToolSpec(
119
+ name="render_chart",
120
+ category="analytics.visualization",
121
+ input_schema={
122
+ "required": ["data", "chart_type", "x", "y"],
123
+ "properties": {
124
+ "data": {"type": "string"},
125
+ "chart_type": {"type": "string"},
126
+ "x": {"type": "string"},
127
+ "y": {"type": "string"},
128
+ "series": {"type": "string"},
129
+ "title": {"type": "string"},
130
+ },
131
+ },
132
+ output_kind="chart",
133
+ description=visualization.DESCRIPTION,
134
+ ),
135
  ]
136
 
137
  # Deferred this round — specs kept intact for easy re-activation, NOT exposed to
src/traceability/scratchpad.py CHANGED
@@ -71,13 +71,40 @@ def _output_to_dict(output: Any) -> dict[str, Any]:
71
  ]
72
  value = getattr(output, "value", None)
73
  if value is not None:
74
- result["value"] = _truncate(value)
 
 
 
 
 
 
 
75
  error = getattr(output, "error", None)
76
  if error is not None:
77
  result["error"] = _truncate(error)
78
  return result
79
 
80
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81
  def _meta_of(output: Any) -> dict[str, Any]:
82
  """Best-effort read of a tool result's `meta` dict (ToolOutput or plain dict)."""
83
  if isinstance(output, dict):
@@ -109,6 +136,11 @@ def _summarize(name: str, out_dict: dict[str, Any], meta: dict[str, Any]) -> str
109
  if name.startswith("analyze_"):
110
  pretty = name.removeprefix("analyze_").replace("_", " ")
111
  return f"Ran {pretty} analysis on {ncol} columns" if ncol else f"Ran {pretty} analysis"
 
 
 
 
 
112
  return name.replace("_", " ")
113
 
114
 
 
71
  ]
72
  value = getattr(output, "value", None)
73
  if value is not None:
74
+ if kind == "chart" and isinstance(value, dict):
75
+ # The full dataeyond.chart.v1 envelope is persisted in message_charts
76
+ # (SPINE_V2_PLAN §4.4); traceability keeps a compact summary — the
77
+ # spec's x/y arrays are numeric, so _truncate's string caps would let
78
+ # them through whole.
79
+ result.update(_chart_summary(value))
80
+ else:
81
+ result["value"] = _truncate(value)
82
  error = getattr(output, "error", None)
83
  if error is not None:
84
  result["error"] = _truncate(error)
85
  return result
86
 
87
 
88
+ def _chart_summary(envelope: dict[str, Any]) -> dict[str, Any]:
89
+ """Compact wire shape for a chart-kind output: chart_type/title/trace_count/
90
+ point_count (charts have no rows, so `row_count` is absent — §4.7)."""
91
+ plotly = envelope.get("plotly")
92
+ traces = plotly.get("data") if isinstance(plotly, dict) else None
93
+ traces = traces if isinstance(traces, list) else []
94
+ points = 0
95
+ for trace in traces:
96
+ if isinstance(trace, dict):
97
+ axis = trace.get("x") if trace.get("x") is not None else trace.get("labels")
98
+ if isinstance(axis, list):
99
+ points += len(axis)
100
+ return {
101
+ "chart_type": envelope.get("chart_type"),
102
+ "title": envelope.get("title"),
103
+ "trace_count": len(traces),
104
+ "point_count": points,
105
+ }
106
+
107
+
108
  def _meta_of(output: Any) -> dict[str, Any]:
109
  """Best-effort read of a tool result's `meta` dict (ToolOutput or plain dict)."""
110
  if isinstance(output, dict):
 
136
  if name.startswith("analyze_"):
137
  pretty = name.removeprefix("analyze_").replace("_", " ")
138
  return f"Ran {pretty} analysis on {ncol} columns" if ncol else f"Ran {pretty} analysis"
139
+ if name == "render_chart":
140
+ ctype = out_dict.get("chart_type") or "chart"
141
+ points = out_dict.get("point_count")
142
+ summary = f"Built a {ctype} chart"
143
+ return f"{summary} ({points} data points)" if points else summary
144
  return name.replace("_", " ")
145
 
146