Commit History
-
[KM-659] fix(eval): make eval a package; run via -m eval.intent.run_eval
Running the file path directly puts only eval/intent/ on sys.path, so 'from src...' fails. Add __init__.py to eval/ and eval/intent/ and document module-mode invocation (uv run python -m eval.intent.run_eval) in the docstring + README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>672926b -
[KM-659] feat(eval): optional Langfuse sink (--langfuse)
Off by default; when set, each case is also sent as a Langfuse trace (grouped under one intent_eval_<ts> session) with an intent_correct 1/0 score, reusing the repo's get_langfuse() client. Token/correctness tracking still lives in the committed results/ JSON — Langfuse is an extra dashboard view, not a dependency. Lazy-imported so the eval runs without langfuse configured. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>9f235a2 -
[KM-659] feat(eval): intent-routing eval runner + metrics
run_eval.py scores the live router per case: correctness (overall + per-intent + per-lang), runtime avg, and token usage (read from the model response via a callback, no Langfuse). Prints a per-case detail table + aggregate summary; writes results/eval_result_<ts>.json (one file per run, never overwritten). README documents the run + the versionable commit convention for results/. pyproject: ignore T201 (print) under eval/ since the runner is a CLI. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>5dcf1ee -
[KM-658] feat(eval): golden intent dataset for the 6-intent router
37 labelled cases, 6-7 distinct scenarios per intent (chat/help/ problem_statement/check/unstructured_flow/structured_flow), balanced EN+ID. carried_over rows mirror old intent_router.md examples for regression; lang enables per-language scoring; id is a stable per-case handle for cross-run diffing. Structured/unstructured rows grounded in real test files; labels decided by phrasing (router has no catalog). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>d52ecfc
