[NOTICKET] planner: surface numeric min/max, so thresholds aren't chosen blind
Rifqi Hafizuddin+1: Claude Opus 5
9155dcb
BUG 14, found by giving analyze_segmentation its first end-to-end run — it was the
last active tool never to have executed outside unit tests.
Asked to band `pa` into low/medium/high, the planner used thirds of 0-100 and all 42
rows landed in "high". A segmentation where every row is in one bucket says nothing,
and nothing in the pipeline flagged it: the tool did exactly what it was told.
The column runs 68.8-95.9. Go has populated stats.min/max on 98 of 98 numeric columns
all along; CatalogSummary dropped them, keeping only sample values. The planner did not
guess badly — it could not see the range.
Same root cause as A28 (constant columns): the catalog knew, the prompt discarded it.
It is also the Python half of the date-range gap handed to Harry — his side populates
dates, this side renders whatever is there, and neither half is useful alone. Date
columns render automatically once he ships.
Rendered as `range=<min>..<max>`, skipped for constant columns because their marker
alrea…
[NOTICKET] assembler: never state the finding a failed step was meant to produce
Rifqi Hafizuddin+1: Claude Opus 5
ec49281
BUG 13, found by the certification run (2026-09-17, analysis 62c34bc6).
analyze_correlation refused a non-numeric column (eqm_no), so nothing was measured.
The reply opened:
"Ya, ada indikasi hubungan antara backlog dan bad actor score..."
(Yes, there is an indication of a relationship...)
assembler.md rule 2 already said "do not pretend they succeeded" AND "do not lead with
the failure". The model obeyed both and invented the conclusion in between. Nothing in
the rule forbade stating the very finding the failed step existed to produce.
This is the dangerous class again: the analysis failed, the narrative claimed a result,
and the hedge arrived two clauses later — where it does not help, because the reader
takes the first clause.
Rule 2 now says so explicitly: not "yes, there is an indication", not "X appears to
drive Y", and not softened with "tends to" or "seems". Name the question that stays
open, and answer only what the successful steps support.
Verified with --turns…
[NOTICKET] eval(chat_sim): --turns, so verifying a late fix doesn't cost a whole suite
Rifqi Hafizuddin+1: Claude Opus 5
0f99ba0
--max-turns only ever takes a PREFIX, so checking a fix to the last turn of an 8-turn
suite meant paying for all eight: ~290k to verify one ~30k turn.
--turns 8 runs exactly that turn. First use, verifying the bug-12 fix: 57,734 tokens
against ~290k for the full suite.
History is still built from whatever runs, so a turn that depends on an earlier one is
selected together with it (--turns 2,8). Noted in the flag's help and in the doc.
Verified on the real suite: turn 8 alone now returns the honest refusal — it names both
tables, says no foreign key is declared between them, and offers the per-table summaries
it CAN produce. No validator vocabulary reaches the user.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
[NOTICKET] planner: exhaustion refuses instead of crashing, + the dry-run findings log
Rifqi Hafizuddin+1: Claude Opus 5
59715a0
BUG 12 — retry exhaustion showed the user the validator's own words.
Asked whether backlog relates to bad actor score, the planner burned all three attempts
(cross-table select, then a join the schema does not declare) and the turn ended with
Analysis failed: planner could not produce a usable plan after 3 attempts;
last error: joins[0]: (...) is not a declared foreign key
That is internal vocabulary about an internal contract, and it reads as a broken product
rather than a limit of the data. The SAME question refused cleanly on the previous run,
so this is reachable by chance, not by input.
Two changes. On the final attempt the planner is now told that refusing is a valid
outcome — until now the retry prompt only ever said "this plan was invalid, fix it",
which pushes toward another plan even when none exists. And if it still exhausts, the
service returns the infeasible sentinel instead of raising, reusing the path that
already handles this well: the coordinator renders it…
[NOTICKET] eval(chat_sim): save a per-run transcript so answers can be audited
Rifqi Hafizuddin+1: Claude Opus 5
0e8f946
The answers ARE the result. Intents, tool names and token counts only say the
pipeline ran; the answer text is the one thing that says it was RIGHT, and the one
thing a person has to read to judge. Until now it existed only in whatever terminal
the run happened in.
That cost a real diagnosis already this week: a `tail` pipe swallowed the early turns
of a run, and the router decision behind a failed turn was simply gone — the re-run to
recover it cost 8.3k tokens and was avoidable.
Each run now writes eval/chat_sim/results/transcript_<suite>_<timestamp>.json holding,
per turn: the question, the router's intent + confidence + rewritten query, the full
answer text, the tools actually called, tokens in/out and latency — plus the run's
source and suite notes, and the report markdown. Timestamped and never overwritten,
matching the convention the other eval harnesses already use for results/.
run_turn now returns a TurnRecord instead of None, and print_turn_steps hands back the
token tota…
[NOTICKET] eval(chat_sim): point the suites at Rifqi's account, unblocking the DB suite
Rifqi Hafizuddin+1: Claude Opus 5
305d629
Rifqi controls 78e8df76 and has registered a database source there: "pa example"
(source_type=schema, 78bf9364) carrying the same four tables — backlog 42, badactor 42,
pa 42, pareto 126. That unblocks the padb suite, which had been written but unrunnable.
This is the first time any dry-run can reach SqlCompiler and DbExecutor at all. Every
fix so far (A26-A31) landed on the tabular path.
Verified before spending anything, with a hand-built IR and no LLM: count -> 42 rows,
and a year/month GROUP BY -> 6 rows with real SQL
SELECT "pa"."year", "pa"."month", AVG("pa"."pa") AS "avg_pa"
FROM "pa" GROUP BY "pa"."year", "pa"."month" LIMIT 10001
so the Fernet credential path decrypts cleanly (no empty-error InvalidToken) and the
compiler emits valid SQL. Noted for the run: the DB path returns numerics as Decimal
where the tabular path returns float — a difference the analyze_* tools have never
been exercised against.
`generated` repointed to this account's copy (d7de686c). `pa` ke…
[NOTICKET] eval(chat_sim): four suites, and a refusal when a source isn't registered
Rifqi Hafizuddin+1: Claude Opus 5
2a6b79e
Adds --suite {pa,generated,padb,cycle}. Each pairs a data source with a question set,
because a question is only meaningful against the shape of the data it runs on: every
floor in the tool layer (_MIN_POINTS, _MIN_BACKTEST_POINTS, _MIN_ROWS) is a statement
about row and period counts, so wording that passes on one source legitimately fails
on another. Each suite records the counts its questions depend on.
pa the existing 7 turns, unchanged (DEFAULT_TURNS now derives from it,
so there is one source of truth rather than two)
generated PA Data Dummy Generated.xlsx, 9,600 rows -- same schema and the same
April 6-25 span as `pa`, different values. Same wording on purpose:
the question is whether A26-A31 were overfitted to one file, and three
of them were found through properties specific to that file, so any
difference in outcome should be attributable to the DATA
padb pa_dummy over Postgres -- the SQL p…
[NOTICKET] eval(chat_sim): mint a message_id per turn, closing the traceability gap
Rifqi Hafizuddin+1: Claude Opus 5
2a435e2
message_traceability and message_charts wrote ZERO rows on every dry-run, and I had
been reporting that as a known harness gap without proving which side was at fault.
Both Python-owned per-turn writes are gated on a message_id: the traceability flush
returns early when `pad.message_id is None`, and chart persistence is guarded by
`if pad is not None and pad.message_id and analysis_id`. POST /api/v2/chat/stream
mints one (`_mint_message_id`); this script called `handler.handle()` without it, so
the parameter defaulted to None and both writes silently no-opped.
So the tables being empty said nothing about the product. It meant chart persistence
and the traceability panel went unexercised on every run — a gap in the evidence
that looked like a result.
Minting one per turn is safe: `message_id` is plain `text NOT NULL` with no foreign
key. Only `analysis_id` references analyses(id), and setup_analysis already creates
that row.
Verified (analysis_id f4471bf1, 2 turns): 2 traceability r…
[NOTICKET] fix(slow path): retry malformed plans, bound stats in the assembler prompt
Rifqi Hafizuddin+1: Claude Opus 5
7a89a2b
Two bugs the 2026-09-17 dry-run surfaced, both found by running the thing rather than
by reading it.
BUG 5 -- a malformed plan crashed the turn instead of being retried.
Structured output came back with task t1 missing `expected_output` and
`success_criteria`. That fails inside `chain.ainvoke` -- pydantic, before
PlannerValidator ever runs -- and the call sat OUTSIDE the try. So this one failure
mode skipped the retry loop every other planner mistake gets, propagated raw, and
reached the user as their literal answer:
ERROR: Analysis failed: 2 validation errors for TaskList
tasks.0.expected_output Field required
A dropped field is exactly as retryable as a bad column_id. The call now sits inside
the loop, and the pydantic message names the missing field, so it is fed back
compacted (`loc: msg`, no docs URLs) and the next attempt is told what to fix. On
exhaustion the caller gets a PlannerError, never a raw pydantic dump.
BUG 6 -- `stats` outputs were rendered to the assemb…
[NOTICKET] fix(router): route data-condition questions to analysis, not inventory (A23 Bug 4)
Rifqi Hafizuddin+1: Claude Opus 5
e815376
"Data PA-nya bersih nggak? Ada kolom yang kosong?" routed to `check` at 0.96
confidence, so the slow path never ran and analyze_data_quality was never called. The
reply was a 36-row schema table whose "Boleh kosong" column is the catalog's declared
`nullable` SCHEMA FLAG -- confident, well formatted, and not an answer to the question
the user asked about the actual rows.
The prompt defined `check` as "what columns are in this table?", and "ada kolom yang
kosong?" is lexically a columns question. The only boundary offered was "what do I
have" -> check versus "analyze/compute/trend/correlate" -> structured_flow, and a
data-CONDITION question is neither phrasing.
Recast the boundary as EXISTENCE vs CONDITION: `check` answers what is there, from
metadata alone; the moment the answer depends on the values in the rows it is
`structured_flow`, because only computing over the data can answer it. Stated
explicitly that naming a column does not make it `check`, and that a `nullable` flag
says …
[NOTICKET] docs: diagnose the missing data-quality turn — a router miss (A23 Bug 4)
Rifqi Hafizuddin+1: Claude Opus 5
0ed7215
Re-ran turns 1-2 alone (analysis_id e3e3ccba, 8.3k tokens), this time writing the log
to a file instead of through a `tail` pipe, which is what cost me the evidence last
time.
The router classified "Data PA-nya bersih nggak? Ada kolom yang kosong?" as `check`,
not `structured_flow`, at 0.96 confidence. The slow path never ran, so
analyze_data_quality was never called and no record was written.
This is a new bug, and the most dangerous of the four, because it does not fail -- it
answers. The reply was a 36-row schema table whose "Boleh kosong" column is the
catalog's declared `nullable` SCHEMA FLAG, reading "—" for 35 of 36 columns. The user
asked an empirical question about the actual rows. The answer is confident, well
formatted, in correct Indonesian, and not an answer to the question -- while 8 of
those columns really do hold one value in every row. Nothing about it looks wrong on
stage.
Cause: intent_router.md defines `check` as "what columns are in this table?", and
"ada kolom …
[NOTICKET] eval(chat_sim): retarget at PA Data Dummy for the A23 dry-run
Rifqi Hafizuddin+1: Claude Opus 5
db7d28f
The Titanic target was gone -- that user's catalog reads back ZERO sources, so the
harness could not have run at all. Retargeted at PA Data Dummy.xlsx (9,729 rows, 36
cols) under the user that owns it, and replaced the scripted turns with the A0 demo
flow, steps 1-7 (step 8, the report, is the capstone the script already runs).
The turns ask for DAILY movement, not monthly, and the reason is recorded in the file
because it is not obvious and will bite whoever reuses this: From_Date spans
2026-04-06..2026-04-25 -- 20 distinct days inside ONE month, verified through the real
query pipeline, not inferred from catalog sample values. At freq="month" the series is
a single period, under analyze_forecast's _MIN_POINTS (4), so a monthly question raises
NotEnoughDataError. At freq="day" there are 20 periods, clearing _MIN_POINTS and
_MIN_BACKTEST_POINTS (8), so the forecast can return a real MASE instead of "not
validated". The A0 flow's "12 bulan terakhir" and "next 6 months" would both have
…
[NOTICKET] planner: group-labelled recipes + the feasibility gate the eval caught (A18)
Rifqi Hafizuddin+1: Claude Opus 5
b56fb4c
Recipes, part 2 of A18. The table goes R1-R6 -> 18 rows, each labelled with its GROUP so
the tool list (organised by group, A17) and the rules (organised by recipe) stop being
two overlapping schemes the planner has to reconcile. New: R7 data-quality, R8 forecast,
R9 importance, R10 pareto, R11 segmentation, R12 clustering, R13 A/B test, R14 anomaly,
R15 method-infeasible. Plus a "telling the close pairs apart" block for the six
intra-group confusions the group hint cannot separate -- trend vs forecast, correlation
vs importance, importance vs pareto, segmentation vs clustering, A/B vs aggregate,
anomaly vs descriptive.
R15 is not a softer R6: R6 is "the column does not exist", R15 is "the column exists but
no tool does that". Numbering note -- the pre-implementation spec called the
method-infeasible recipe R12; the implemented table gives R12 to clustering and R15 to
method-infeasible. Stale references in DEV_PLAN, examples.py and groups.py corrected
rather than renumbering the live …
A0 activated every analytics tool, which puts the planner-visible set at 18 -- inside
the measured 15-20 band where tool-selection accuracy degrades, and this registry has
exactly the semantic near-duplicates that drive it (trend/forecast,
correlation/importance, segmentation/cluster). A0 chose to pay that cost with grouping
rather than by turning tools off, so nothing goes dark.
Three layers, and which does what is the whole design:
router (LLM, already runs) -> emits group_hint on its existing structured output
-> planner/groups.py (deterministic, 0 LLM) -> filters the registry
-> planner (LLM) -> plans against the slice, unchanged
The filter is deliberately NOT the planner's job: to choose a group it would have to
be shown every group, which is the thing being avoided, and a second LLM call would
break the two-call invariant for a benefit that only pays above ~60 tools. It is not
the router's job either -- the router classifies, it does not filter.
Groups are DERIVED…
30 days ago
[NOTICKET] planner: rebalance few-shot exemplars EN/ID, demo the reply-language rule
Rifqi Hafizuddin+1: Claude Opus 5
fb30aec
A19. The exemplar set was 13/13 English while the eval dataset -- and real traffic --
are majority Indonesian (17 ID / 14 EN). In-context learning over-predicts whatever
the examples show most (majority-label bias, Zhao et al. ICML 2021), so an all-English
set teaches the planner that Indonesian questions are out-of-distribution.
Converted four questions to Indonesian (now 9 EN / 4 ID), chosen so each conversion
also breaks a near-duplicate:
D breaks the 3x check_data -> retrieve_data -> analyze_aggregate stack (A/D/E)
M stops reading as a near-duplicate of J's English bar-chart ask
L R2b "berapa banyak ..." is the recipe most often asked in Indonesian
H the infeasible example -- see below
H now demonstrates the reply-language rule: an Indonesian question gets an Indonesian
infeasible_reason while every other field stays English. prompt.py:151 has always
mandated this ("If (and only if) you return an infeasible plan, write
infeasible_reason in {reply_language}") but NO e…
A16. Moving a tool to DEFERRED_ANALYTICS_TOOLS turns its eval cases red, and that
noise hides a real planner regression in the same run. Those cases are now reported
[SKIP] and leave the denominator, so a score stays comparable to a run taken when a
different set of tools was active.
The requirement is DERIVED from each case's own assertions rather than a hand-kept
field, so the dataset stays self-maintaining:
has_tool: X -> skipped when X is not planner-visible
any_tool: [...] -> skipped only when every listed tool is withheld
no_tool: X -> asserts absence, never a reason to skip
Cases stay in planner_dataset.json and reactivate the moment the tool returns to
ACTIVE. The skip is checked before the LLM call, so a withheld tool costs no tokens.
--rescore carries a saved run's skips through unchanged.
Verified without spending a run:
- 0 skips against today's 10-tool registry (a no-op until A15 lands)
- deferring analyze_correlation skips exactly corr_mttr_pa_id + c…
