Commit History
-
[NOTICKET] fix(tools): accept `avg` wherever `mean` works (A23 Bug 2)
Rifqi Hafizuddin +1: Claude Opus 5The planner and the tools disagreed about what "average" is called, and the planner lost. It fills in a QueryIR and a tool call from ONE prompt: QueryIR AggFn count, count_distinct, sum, avg, min, max tools SUPPORTED_AGGS sum, mean, count, min, max, median (+ nunique) planner.md teaches `avg` three times because the IR requires it -- line 331 even says "utilization, scores -> avg", which is exactly the PA case. So the planner wrote agg="avg" into analyze_forecast and the tool raised UnsupportedAggregationError. The dry-run lost its forecast turn that way. The planner was not wrong; it obeyed its own prompt. Blast radius was five tools, not one. analyze_trend, analyze_forecast, analyze_ab_test, analyze_pareto and analyze_segmentation all take an `agg` and all five rejected `avg` -- only forecast happened to be exercised. analyze_aggregate failed differently and more quietly: validator check 8c rejected the plan, burning a retry before execution. Aliasing rath…0f0af11 -
[NOTICKET] fix(tools): analyze_ab_test quality — KeyError on the too-few-rows path
The quality block (added in 2082eba1) read test['p_value'] directly, but _welch_test's too-few-rows branch returns no 'p_value' key -> KeyError, so analyze_ab_test crashed on a valid 'too few rows for a t-test' case (became an error envelope via the never-throw seam instead of a graceful result). Fix: test.get('p_value'). The too-few case now returns significant=None + quality {value: None, weak: True}. Caught by a local ab_test test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>8400a9e -
[NOTICKET] tools: quality blocks for forecast + ab_test, forecast backtest, CK7 fix
Rifqi Hafizuddin +1: Claude Opus 5Extends the reserved `quality` convention to the tools where it means something, and fixes a CK7 message that was wrong for half its inputs. analyze_forecast -- the substantial one. It reported NO accuracy at all, so "how accurate is it?" had no answer. Now backtests: hold out the last h periods (bounded by a quarter of the series and by the fitting floor), refit, and score with MASE -- mean absolute error over the naive "next = last" error on the training span. MASE is scale-free (Hyndman & Koehler 2006) and, unlike MAPE, neither explodes near zero nor punishes over-prediction asymmetrically. Both fits run in ONE run_fit call, so a forecast still costs one thread hop and one timeout. A series too short to hold anything out, or one that never moves, ships UNVALIDATED -- also weak, with the two causes reported as distinct reasons rather than one wrong one. analyze_ab_test -- free and exact: the statistical verdict IS the quality verdict. A gap the test cannot separate from noise is we…2082eba
-
[NOTICKET] feat(tools): analyze_ab_test — rename analyze_comparison + Welch t-test (A11)
Renames analyze_comparison/comparison.py -> analyze_ab_test/ab_test.py. Keeps the two-group aggregate gap; adds a Welch two-sample t-test (t, p-value, significant at alpha) on the raw values and a bar chart of the two group means with 95% error bars, via the reserved 'charts' key. Registered DEFERRED via spec_from_model. Old name removed — 5 cross-ref DESCRIPTIONs + contracts.py comment repointed. DEV_PLAN §0.9 A11 -> done. Heads-up (Rifqi): analyze_comparison renamed to analyze_ab_test. Local tool tests under tests/unit/tools (gitignored) referencing it break on pull — rename to analyze_ab_test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>901b03f