Commit History
-
[NOTICKET] fix(eval): don't let a crashed help case pass its rules (flag errored)
Review follow-up on eval/help: if HelpAgent.astream raised, output became "ERROR:..." and assertions still ran on it — must_not_contain_any trivially passed and language_match fell back to Indonesian, so an infra failure could inflate compliance. Now such a case is flagged `errored`, its rules are not scored, it counts as a failure, and it's surfaced under a separate ERRORED line (not mistaken for a rule violation). Also: type _build_state -> AnalysisState (mypy strict), and document the known limitations (temperature variance, detector-graded language, errored handling) in the README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>760ba69 -
[NOTICKET] fix(eval): update readiness eval to post-KM-652 floor (drop problem_validated)
The readiness eval crashed at import (`_MISSING_PROBLEM` gone) and would then have hit a TypeError (`stub_analysis_state(problem_validated=...)` — arg dropped in #4). Same #4 rework regression as test_help. Aligned it to the current floor (>=1 substantive record; delta-since-report), which no longer gates on a validated goal: - run_eval: drop the `_MISSING_PROBLEM` import + the `problem` -> code mapping; call stub_analysis_state() with no arg. - dataset: remove the `problem_validated` field and the `problem` missing-code from every case; drop the two cases that only existed to isolate the retired problem gate (now duplicates); renumber the floor group. Runs 15/15 floor-correct, alignment gap = 2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>6f46e81 -
[NOTICKET] feat(eval): add help-skill eval harness + goal-based language fallback
Live-model eval for the Help skill (eval/help), mirroring eval/intent + eval/readiness: each case declares state + report_ready + history, streams HelpAgent.astream for real, and scores RULE compliance (reply language; never/always suggest a report per report_ready) — not text similarity. Held-out vs carried_over reported separately so prompt overfitting is visible; orientation cases are manual_review. help.py: reply-language detection now consults the user-authored goal (objective + business_questions) when there is no chat turn to read, before the hard Indonesian fallback. The goal is required at onboarding, so on a fresh analysis it is a reliable signal — a user whose goal is in English no longer gets an Indonesian reply. Priority: message > last human turn > goal > default. help.md v3: language rule promoted to a hard [Reply language] directive; Examples given stable ids so the eval can mirror them as carried_over regression cases; second example now uses a real is_report_ready `…cbdbefb
-
feat/Knowledge & Data Tools (#3)
- [KM-641] refactor(tools): trim analytics registry to 4 active tools (66e2e4df66f1abf7eaa7e442d9900d428696ece4) - [NOTICKET] docs: add checkpoint plan for 2026-06-17 sync (8c1bb63ebdee02f9db0da9fc2b2269823b0a7688) - [KM-642] refactor(tools): rename data-access tools to data/knowledge naming (c38c0c29d3e6b245f67d03f2c35e451900e1abdb) - [KM-643] feat(tools): merge data sources into data_check + add knowledge_check (4bd5f1efdc560a64092cc8df672a7990fa949b59) - [NOTICKET] docs: record KM-642/643 in checkpoint plan + PROGRESS (4c57588b90fe0cac8d0d4d22cfe5701fc9eb52dd) - [NOTICKET] fix(chat): scope response cache by user_id + drop TTL to 1h (R5) (b701e95d300c7d2480a3608b2d4da6dfe102b19a) - [NOTICKET] feat(slow-path): persist AnalysisRecord via PostgresAnalysisStore (09f845188226dd01633cf74525caa569e4a7d37a) - [NOTICKET] Adopt verb-first skill naming (2d6406def7ed45edd1226e96645ba2a5cdbfa3d0) - [NOTICKET] docs: sync checkpoint plan — verb-first tool names + report-as-API (3b399f84d73b0d98bd99…0721bb4
