Commit History
-
[KM-668] feat(eval): model tasks_run; substantive = successful analyze (T2)
Update the readiness eval for the KM-652 T1 fix (substantive = a successful analyze_* task, not merely 'has findings'). Fake records now carry tasks_run (_FakeTask: status + tools_used); dataset records use analysis=success|failure|none. New cases: floor_04 (analyze FAILED but findings present — the live e2e bug), floor_05 (data-access-only, no analyze — the 'any task succeeded' trap), delta_05 (newer analysis is a failure — failed retry must not unlock a duplicate report). Floor 17/17 vs the fixed readiness.py; alignment gap still 2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>b1dd8e6 -
[NOTICKET] feat(eval): report-readiness eval (dataset + runner + baseline)
Deterministic eval for is_report_ready (no LLM, no DB — fake stores). 16 golden cases across floor / delta / edge / alignment; scores both the ready boolean and the missing gaps, and counts the alignment gap (floor says ready but analyses unaligned to the problem statement). Baseline run: floor 16/16 (100%), alignment gap = 2 (align_01, align_02) — the evidence for/against the deferred LLM-judge. Run: uv run python -m eval.readiness.run_eval Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>990f264
-
[KM-659] fix(eval): make eval a package; run via -m eval.intent.run_eval
Running the file path directly puts only eval/intent/ on sys.path, so 'from src...' fails. Add __init__.py to eval/ and eval/intent/ and document module-mode invocation (uv run python -m eval.intent.run_eval) in the docstring + README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>672926b -
[KM-659] feat(eval): optional Langfuse sink (--langfuse)
Off by default; when set, each case is also sent as a Langfuse trace (grouped under one intent_eval_<ts> session) with an intent_correct 1/0 score, reusing the repo's get_langfuse() client. Token/correctness tracking still lives in the committed results/ JSON — Langfuse is an extra dashboard view, not a dependency. Lazy-imported so the eval runs without langfuse configured. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>9f235a2 -
[KM-659] feat(eval): intent-routing eval runner + metrics
run_eval.py scores the live router per case: correctness (overall + per-intent + per-lang), runtime avg, and token usage (read from the model response via a callback, no Langfuse). Prints a per-case detail table + aggregate summary; writes results/eval_result_<ts>.json (one file per run, never overwritten). README documents the run + the versionable commit convention for results/. pyproject: ignore T201 (print) under eval/ since the runner is a CLI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>5dcf1ee -
[KM-658] feat(eval): golden intent dataset for the 6-intent router
37 labelled cases, 6-7 distinct scenarios per intent (chat/help/ problem_statement/check/unstructured_flow/structured_flow), balanced EN+ID. carried_over rows mirror old intent_router.md examples for regression; lang enables per-language scoring; id is a stable per-case handle for cross-run diffing. Structured/unstructured rows grounded in real test files; labels decided by phrasing (router has no catalog). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>d52ecfc