| # Chat simulator (end-to-end, in-process) |
|
|
| Drives the **real** `ChatHandler.handle()` in-process to simulate a user chatting in |
| one analysis session — from creation through a report — and prints **what each step |
| does**: router decision, slow-path status, every tool call, every LLM call (output + |
| tokens + latency), the streamed answer, and a final report built from the run's |
| `report_inputs`. |
|
|
| In-process (not via the HTTP server) on purpose: the SSE endpoint hides the internal |
| LLM outputs (router/planner/assembler), so a script that consumes `/chat/stream` can't |
| show them. This hooks the same tracer seam the handler threads its Langfuse callbacks |
| + tool spans through (`ScriptTracer.active=True`) — **no source changes**. |
|
|
| ## Run |
|
|
| Module mode (`-m`) so `src` imports resolve; needs a populated `.env` (Azure OpenAI + |
| Postgres + Azure Blob for the Titanic Parquet). `ENABLE_SLOW_PATH` is forced on here. |
|
|
| ```bash |
| uv run python -m eval.chat_sim.run_chat # scripted Titanic convo + report |
| uv run python -m eval.chat_sim.run_chat --interactive # type your own messages; 'report' / 'exit' |
| uv run python -m eval.chat_sim.run_chat --max-turns 1 --no-report # cheap smoke test |
| uv run python -m eval.chat_sim.run_chat --no-bind # planner sees the whole catalog (not just Titanic) |
| uv run python -m eval.chat_sim.run_chat --plain # no ANSI colors |
| ``` |
|
|
| (Or `./.venv/Scripts/python.exe -m eval.chat_sim.run_chat ...` on Windows.) |
|
|
| ## What it does each run |
|
|
| 1. **Creates a fresh analysis** (`AnalysisStateStore.create`) with an objective, and |
| **scopes it to the Titanic source** by seeding an analysis-scope `data_catalog` row |
| (the user catalog restricted to Titanic) so `structured_flow` is scoped to one |
| source — same as `/analysis/create` (in production Go materializes this row from |
| `analyses.data_bind`). |
| 2. **Runs each turn** through `handle()` and prints the router decision, slow-path |
| status pings, the tool table (kind / rows / latency / error), the LLM table |
| (in/out/total tokens + ms + a prompt/output snippet), and the answer + sources. |
| 3. **Generates a report** (mirrors `POST /report`: floor check → `ReportGenerator` → |
| `ReportStore`) from the `report_inputs` the structured turns persisted. |
|
|
| ## Notes |
|
|
| - **Default user** is `4b5d1bac-…` whose playground catalog has the Titanic CSV |
| (tabular) + a "dummy" Postgres (schema). Override with `--user-id`. |
| - **Writes to the DB the `.env` points at** — a fresh `analyses` row + `report_inputs` |
| + a `reports` row per run. Point `.env` at the playground DB. Each run is a new |
| `analysis_id` (printed at the end) so runs don't collide. |
| - **"output: <no text content — structured / tool-call output>"** is expected for the |
| router, planner, and assembler — they use structured/function-call output, so the |
| LLM message has no plain text. Their result shows up in the ROUTER line, the plan, |
| and the streamed ANSWER respectively. The chatbot/help calls stream text, so their |
| output is shown (annotated `masked→cloud` where the cloud trace would redact it). |
| - The first scripted question may route to `chat` rather than `help` — that's the live |
| router's real call, shown transparently. |
|
|
