Commit History
-
[NOTICKET] fix(eval): don't let a crashed help case pass its rules (flag errored)
Review follow-up on eval/help: if HelpAgent.astream raised, output became "ERROR:..." and assertions still ran on it — must_not_contain_any trivially passed and language_match fell back to Indonesian, so an infra failure could inflate compliance. Now such a case is flagged `errored`, its rules are not scored, it counts as a failure, and it's surfaced under a separate ERRORED line (not mistaken for a rule violation). Also: type _build_state -> AnalysisState (mypy strict), and document the known limitations (temperature variance, detector-graded language, errored handling) in the README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>760ba69 -
[NOTICKET] feat(eval): add help-skill eval harness + goal-based language fallback
Live-model eval for the Help skill (eval/help), mirroring eval/intent + eval/readiness: each case declares state + report_ready + history, streams HelpAgent.astream for real, and scores RULE compliance (reply language; never/always suggest a report per report_ready) — not text similarity. Held-out vs carried_over reported separately so prompt overfitting is visible; orientation cases are manual_review. help.py: reply-language detection now consults the user-authored goal (objective + business_questions) when there is no chat turn to read, before the hard Indonesian fallback. The goal is required at onboarding, so on a fresh analysis it is a reliable signal — a user whose goal is in English no longer gets an Indonesian reply. Priority: message > last human turn > goal > default. help.md v3: language rule promoted to a hard [Reply language] directive; Examples given stable ids so the eval can mirror them as carried_over regression cases; second example now uses a real is_report_ready `…cbdbefb
