Spaces:
Sleeping
Sleeping
Commit History
-
Deploy: dashboard naive-vs-rigorous reveal + sync serving to origin/main
Single squashed deploy commit on top of hf/main. Brings the reveal (source-gated reveal_anchor.json, render-time injection, progressive- enhancement JS) and the accumulated origin/main serving/stats/docs work. HF frontmatter + Dockerfile (port 7860) preserved; README refreshed to the repositioned framing. Plot PNGs excluded (HF Xet policy; they live on GitHub and serve the README card there, not the running app).1c08abf
-
calibrate(jury): v1.1+v1.1.1 — fix weighting bugs; recency-position paraphrase clause
Three nested findings on the judge layer's calibration set, all discoverable from `docs/_generated/kappa_table.md` joined to the per- member sidecar at `results/...jury_kappa_weighted_*_members.jsonl`. Full mechanism + v1.2 escalations in DECISIONS.md "v1.1 jury rescue" and "Plan 3A" entries. 1. v1.1 jury rescue (κ 0.014 → 0.416 on completeness, no API spend). Two compounding v1 bugs forced equal weights and let gpt-4o-mini's downward bias win every disputed cell: `_load_weights_from_baseline` was a stub returning 1.0 from a Haiku-only source, and `jury.py`'s missing-key fallback returned 1.0 for any member not in the source. v1.1 replaces the stub with `_compute_kappa_weights` (real per-judge κ vs gold, hard-error on asymmetric coverage, negative-κ clipped to 0) and converts the missing-weight fallback to a `ValueError`. Re- aggregation of the existing 164 sidecar rows with corrected weights recovers the Haiku-baseline κ exactly. Pre-committed Outcome 2 …ab0e054
-
rubric: clarify groundedness reference scope (snippets-only) for v1.1 gold (#20)
Jane Yeung +1: Claude Opus 4.7 (1M context)Unverified signature* rubric: clarify groundedness reference scope (snippets-only) for v1.1 gold The Opus stress-test pass disagreed with the v1.0 single-rater human gold on 22 of 30 groundedness items, all in the same direction (human=1, opus=0). The pattern traced to a reference-scope drift: rubric scopes groundedness to the gold snippets only, but the v1.0 labels were calibrated against the broader corpus. Strict-snippet entailment is methodologically load-bearing — it measures retrieval-grounded behavior, not LLM general knowledge passing through a RAG harness — so the rubric holds and the labels move. This commit lands three coupled artifacts: 1. Rubric clarification (agent_bench/evaluation/rubrics/groundedness.md). Adds an explicit reference-scope section with normative "must score 0", a trivial-inference clause covering paraphrase + unit conversion + syntactic variation + canonical-name carve-out, and three calibration anchors (k8s_006 dramatic over-extension, q006 subtle embellishme…e16544c
-
fix(types): four mypy errors blocking CI
I ran ruff locally after each fix batch but skipped mypy (the project's make lint runs all three; I only invoked the ruff portion). CI caught what I should have. #1 harness.py:200 — _JUDGE_CLASS_BY_DIMENSION was inferred as dict[str, type[Judge]], so the dispatch site looked like an attempted instantiation of an abstract class. Annotated with the explicit union of concrete subclasses so mypy sees instantiability. #2 report.py:64 — abstain filter doesn't narrow ScoreResult.score from `int | Literal["Unknown"]` to int for mypy. Cast through float() with a targeted `# type: ignore[arg-type]` since the runtime guarantee (abstained=False → score is int) isn't expressible in the type system without overloads. #3 rubric_permute.py:75 — local `score: int | str = "Unknown"` should be `int | Literal["Unknown"]` to match ScoreResult.score's signature. #4 citation_faithfulness.py:169 — same fix: aggregate_score typed as `int | Literal["Unknown"]` instead of `int | str`. mypy now clean; 523 te…02b8717 -
docs(harness,readme): two re-review must-fix items
harness.py:84 — EvalResult.judge_scores docstring was stale: said 'Empty when item.category == "out_of_scope"', no longer true after the per-dimension OOS gate fix. Updated to describe the actual behavior: OOS items get relevance only, reference-based dims (groundedness, completeness) skipped on OOS, completeness skipped when reference_answer is empty regardless of category. README.md — test count 509 → 523 (caught up with the +9 review regression tests + the +5 Codex regression tests), and the evaluate-full cost note widened from $0.01–0.05 to $0.01–0.10 with an explicit per-item breakdown so the OOS-gets-relevance change doesn't quietly outgrow the upper bound. The change is small for the current dataset (~10% OOS × 1 extra judge call) but the cost note should reflect the cost model, not just one dataset's empirical floor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>c39d5c7 -
fix(judges,calibration,harness): three Codex adversarial-review findings
Codex Fix #1 — Single-row ablation options now actually take effect. Pre-fix, scripts/run_calibration.py never read row['options'], so baseline_no_cot, baseline_no_anchors, and baseline_no_abstain produced identical κ to baseline. The κ ablation table would have been methodologically invalid. Three knobs threaded through: - use_cot: Judge ABC accepts use_cot=True default; concrete judges emit a JSON schema clause via _json_schema_clause() helper that drops 'reasoning' and 'evidence_quotes' when False. The helper parser tolerates missing 'reasoning' key (data.get default '') so no_cot responses don't fail validation. - use_anchors: implemented via Rubric.strip_anchors() returning a copy with ### Example sections removed. body_markdown changes → source_hash naturally diverges → ScoreResult.rubric_version bucketing is automatic. - abstain_allowed: Judge ABC accepts abstain_allowed_override=None; effective_abstain_allowed property returns override when set, else rubric.a…226b6f4 -
fix(judges,calibration): five review follow-ups (items 5, 6, 7, 9, 10)
#5 — Gwet's AC2 weighted variant now raises NotImplementedError. The unweighted AC1 formula is correct and tested against three hand- computed cases. The weighted variant has multiple inconsistent literature definitions (Gwet 2008 vs 2014) and no sklearn analogue to cross-check, so shipping a plausible-looking weighted formula without a fixture is a methodology hazard. Gate it explicitly until v1.1 pins both formula choice and fixture. test_weighted_variant_ raises_not_implemented covers both 'linear' and 'quadratic' kwargs. #6 — citation_faithfulness empty-claim handling. When the answer starts with a [source:] citation (no prior content), the extractor returns an empty claim string. Previously the judge built a prompt with empty content and burned an API call asking the LLM to evaluate emptiness. Now: vacuously faithful (score=1, no API call), with a synthetic ScoreResult so per-pair detail still appears in evidence_quotes. test_leading_citation_empty_claim_vacuously_ faithful asser…71ec5e8 -
fix(judges): four review-blocking bugs (review items 1–4 + 8)
#1 — harness.py CompletenessJudge gate restored. Pre-supersession code gated correctness on `if q.reference_answer:`; the new per-dimension loop iterated all dims unconditionally, burning tokens on guaranteed- noisy verdicts when reference_answer was empty. Now: skip completeness when reference_answer is falsy, matching the prior contract. Test asserts the gate by mocking a judge_provider and confirming 'completeness' is absent from judge_scores when reference is "". #2 — Rubric loader was fence-blind. `## Score N` literals inside fenced code blocks in anchored examples were counted as structural level headers, producing arity-mismatch errors on rubrics that wanted to quote header-shaped strings (which the design encourages). Fix: mask fenced regions with same-length whitespace before scanning for level headers, then slice the original body at the masked-text header positions to recover level bodies with their fenced content intact. New fixture rubrics_valid_with_fenced_examples.md ex…9255fb5 -
refactor(metrics): delete superseded LLM judges (answer_faithfulness etc.)
Removes: - _FAITHFULNESS_PROMPT, _CORRECTNESS_PROMPT (string constants) - answer_faithfulness(), answer_correctness() (continuous 0–1 judges) - _judge_call() (silent-None error handler that obscured failures — the new layer's abstain-as-ScoreResult discipline replaces it) - json + Message + Role imports (unused after deletion) - LLMProvider import (unused after deletion) Keeps all deterministic metrics (retrieval_precision_at_k, keyword_hit_rate, citation_accuracy, etc.) — the L1 layer survives unchanged. Module docstring rewritten to point at the new judge package and the design doc that motivated the supersession. Hard cut, no deprecation. The 5 YAML configs already preserve judge_provider; the harness migration in the previous commit rewires through agent_bench.evaluation.judges. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>281b43d -
refactor(harness): migrate to per-dimension Judge layer (drop faithfulness/correctness)
Hard-cut supersession: EvalResult loses faithfulness + correctness fields, gains judge_scores: dict[str, ScoreResult]. The optional L2 block in run_evaluation now iterates evaluation.judge_dimensions from config and dispatches per-dimension Judge instances built from the rubric markdown files at agent_bench/evaluation/rubrics/. The judge_provider != None gate is preserved (existing harness behavior); the q.category != 'out_of_scope' gate is preserved (L2 doesn't apply to refusals — that's L1's job). ScoreResult is imported at module level (not under TYPE_CHECKING) because pydantic needs the runtime symbol to validate the dict[str, ScoreResult] field — TYPE_CHECKING-only forward refs crash with PydanticUserError on first EvalResult instantiation. The reverse direction (judges/base.py importing harness types) is still TYPE_CHECKING-only, so no runtime cycle. evaluation/report.py rewires from the old faithfulness average to a groundedness-judge average computed from r.judge_scores['gro…e76227f -
feat(calibration): generate_kappa_table with strict/warn modes
Joins predictions ⋈ labels by (item_id, dimension, system_output_hash). Hash mismatch ALWAYS raises with first-item expected/actual hashes plus full mismatched-id list — applies in both modes (never warned). Missing predictions/labels warn-and-exclude by default; --strict raises (the final-artifact path; make calibrate uses it). Pairwise abstain exclusion in κ; per-dimension cause breakdown (schema_parse / out_of_range / provider_exhausted / genuine) via the abstain-reason constants from judges/base.py. Abstain-rate flag fires on STRICTLY greater than 20%; 6/30 (=20%) does not fire, 7/30 does — boundary tested explicitly. κ undefined → '—' with footnote (insufficient variance, N<3 agreement-eligible items remaining, or all labels+predictions in a single category — the last condition was load-bearing for the 'all-ones degenerate' test case). Skips '_members.jsonl' sidecar files in the predictions glob — they hold per-permutation / per-jury-member detail, not aggregate predictions. C…1d47106 -
feat(scripts): run_calibration.py orchestrator for Steps A/C/D
Three subcommands, all sharing concurrency-resolution + structured logging: generate-outputs — Step A: orchestrator against 30 calibration items, frozen config, writes results/calibration_v1_system_outputs.json run-judges — Step C: takes --row-config=<path>, scores frozen outputs with that row's judges, writes results/calibration_v1_judge_<label>.json build-table — Step D: invokes generate_kappa_table; --strict raises on missing predictions/labels Resolved concurrency value logged at every run so artifacts capture which concurrency was used. Default 5; CLI overrides config-field fallback overrides hardcoded default. Step B (hand-labeling) is manual — done in a Jupyter notebook, not orchestrated by this script. Also folded in lint fixes for the Phase 1-3 modules to satisfy ruff E402 (test imports moved to top of test_judges.py) and E501 (jury.py reasoning …4fa7c61 -
feat(goldens): add source_snippets to 8 FastAPI calibration items
Hand-snippeted verbatim spans from data/tech_docs/ for the 8 FastAPI items in calibration_v1. OOS items get source_snippets:[] (no source to ground against). Scope discipline: only the 8 calibration items, not the full 27-item FastAPI golden — the remaining 19 backfill in v1.1. Required by GroundednessJudge (reference-based on source_snippets). K8s items already had this field from the multi-corpus refactor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>a48afb9 -
feat(calibration): 30-item stratified calibration_v1 sample
Stratified across FastAPI (categorized) + K8s (CRAG question_types) per the design doc's sampling table. 26 items from explicit strata + 4 spare slots from K8s simple_w_condition / multi_hop / comparison / false_premise (highest-variance R@5 strata in pre-judge runs). Sample seed 20260504 (date-derived) so the sampling is reproducible. The spare pool was widened from {simple_w_condition, multi_hop} to also include {comparison, false_premise} because the K8s golden set holds only 4 simple_w_condition and 6 multi_hop items, of which the explicit targets already consumed 7, leaving 3 in the original pool — one short of the 4 spares the design's 30-item total requires. system_config_git_sha pins the commit producing the sample; the soon- to-be-generated system_outputs file references this SHA so any drift in the pipeline between sample generation and output generation is detectable in the calibration report. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>8ef480a -
feat(calibration): hand-rolled cohen_kappa, gwets_ac2, bootstrap_ci
Hand-rolled (not sklearn) per the design's 'built from primitives' discipline. Cohen's κ: (P_o - P_e) / (1 - P_e), supports unweighted, linear, and quadratic weight matrices for ordinal scales. Gwet's AC1/AC2: chance term = (1/(q-1)) Σ pi_k(1-pi_k) on mean marginals, more robust to skewed distributions than Cohen's κ. Bootstrap CI: seed=42 for reproducibility, drops degenerate resamples (all-one- label) silently rather than raising. Three hand-computed test cases per metric (perfect agreement κ=1, complete disagreement κ=-1, chance agreement κ=0) include worked- out arithmetic in comments so a reader can verify formula correctness without running the test. sklearn-parity scaffold included with placeholder fixtures and a skip until scripts/_dev/generate_kappa_fixtures.py runs in a venv with sklearn==1.5.2; the cross-check test is also gated on the JSON sidecar existing. Phase 4 Task 4.2 lands the generator. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>6ef2e0e -
feat(variance): PermutedJudge + Jury — N permutations and multi-judge aggregator
PermutedJudge wraps a Judge and runs N permutations with different prompt_seeds. Discretization: binary thresholds at 0.5 with ties → 0 (conservative); three-point rounds to nearest with ties → lower level. Any abstain in any permutation propagates to Unknown — the variance signal that permutation is designed to surface should not be averaged away. Per-permutation ScoreResults written to sidecar JSONL (append- mode so multiple score() calls across items accumulate). Jury aggregates a list of Judge instances via asyncio.gather with return_exceptions=False so non-retryable exceptions cancel sibling tasks immediately (failing fast on caller bugs). Per-member ScoreResults written to sidecar — successes AND failure-as-abstains — so the calibration report can compute per-judge κ even when the aggregate row drops to abstain via the quorum gate. Strict quorum default (quorum = len(judges)) at v1's 2-judge jury means any member abstain → jury abstain. Tolerant defaults at N=2 are silent singl…c038a7d -
feat(judges): CitationFaithfulnessJudge with all-or-nothing aggregation
Per-(claim, citation) binary judge that aggregates to one item-level ScoreResult via all-or-nothing — any unfaithful citation → score=0. Per-pair detail preserved in evidence_quotes. Opt-in for v1 (judge_dimensions default excludes it); default-on in v1.1 once the citation-deterministic-vs-LLM head-to-head section of the writeup validates the gain over the existing regex-based citation_accuracy. Any sub-call abstain propagates to the aggregate (consistent with the strict-quorum / any-abstain principles in jury and rubric_permute). _extract_claims_with_citations strips prior [source:] tags from the left context before searching for sentence boundaries, so multi- citation answers yield clean claim strings rather than carrying prior citation tags into downstream claims. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>04d9ea0 -
feat(judges): CompletenessJudge + three-point reference-based rubric
Three-point rubric (none / partial / full) scored against the gold reference_answer. Coverage-of-facts framing: score only on what fraction of the reference's points are present, not on additional correct facts. Two anchored examples per level. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>80be2d8 -
feat(judges): RelevanceJudge + anchored three-point rubric
Three-point rubric, reference-free (judge sees only question + answer, not gold snippets or reference answer). Score the topic-match, not the truth-value. Two anchored examples per level (six total) covering wrong-topic, incomplete-engagement-refusal, adjacent-but-off-target, sibling- question-answer, and on-target single-fact + comparison cases. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>b170eb6 -
feat(judges): GroundednessJudge + anchored binary rubric
Binary rubric, reference-based on item.source_snippets. Strict- conjunction definition: any unsupported claim fails the rubric. Two anchored examples per score level with thinking-trace explanations (per Singla et al. 2025 — through-the-judge's-eyes). Per the design doc's sequencing tactic, groundedness is authored first and dry-fitted before relevance and completeness — this converts rubric authoring from three parallel risky tasks to one risky task plus two near-mechanical replications. system_output_hash is SHA-256 of (item.id, answer, sorted(sources)) joined by NUL; the calibration report uses this as the agreement- eligibility key against labels. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>30a5e0c -
feat(judges): _call_judge_with_retry helper with strict-reprompt + abstain
Module-level helper (not a Judge method) called by every concrete judge. One retry with the strict-formatting reprompt suffix on schema-parse / score-out-of-range; abstain with structured-prefix reason on persistent failure or provider exhaustion. Caller bugs (401, 400, anything not in the retryable taxonomy) propagate. The first-attempt-failure WARN log fires regardless of whether the retry succeeds — the silent recover-on-retry case is the most analytically interesting one for offline analysis (tells you the reprompt is doing work, not just costing 2x tokens). Test uses structlog.testing.capture_logs (not pytest caplog) since the project does not configure structlog to route through stdlib logging — capture_logs is the structlog-native test seam. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>ff78845 -
feat(judges): MockJudge with LookupError on missing keys
MockJudge raises LookupError (not a default) on missing item.id keys, so test fixtures are self-checking against rename drift. A separate fixture-validation test in Phase 8 walks item.id across all goldens and asserts coverage; the LookupError is the second layer of defense. __init__.py re-exports the public surface for ergonomic imports (from agent_bench.evaluation.judges import Judge, ScoreResult, ...). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>aa70e89 -
feat(judges): Judge ABC with judge_id derived from model + dimension
Judge is abstract — concrete subclasses (groundedness, relevance, completeness, citation_faithfulness) land in Phase 2 as thin ~30-line classes per the no-shared-base-method discipline. judge_id format: '{model_id}_{rubric.dimension}', e.g. 'claude-haiku-4-5_groundedness'. The format is load-bearing for the calibration report's per-judge κ breakdown — the report groups by judge_id when computing per-judge agreement against the human labels. Cross-package types (GoldenQuestion, AgentResponse, LLMProvider) are imported under TYPE_CHECKING to keep judges/base.py loadable before harness.py is migrated in Phase 8 (avoiding a circular import once harness imports back from judges). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>2192305 -
feat(judges): Rubric markdown loader with aggressive validation
Rubric loads from markdown with YAML frontmatter; validates scale, arity-matches-scale, anchored-example-per-level, frontmatter required fields. ValidationError raises with file path + field context so malformed rubrics fail at construction (Day 1) not at first judge.score call (Day 2 with API budget spent). source_hash is SHA-256 of body_markdown — immutable per file content, independent of git state. Used as ScoreResult.rubric_version so κ aggregation can group by rubric identity without cross- referencing run metadata. render_prompt(level_permutation_seed=N) deterministically permutes the ## Score sections via seeded PRNG. Seed=0 returns canonical order; this is the variance-control hook used by rubric_permute in Phase 3. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>7b72b2c -
feat(judges): ScoreResult + abstain-reason constants
ScoreResult is the per-call record; field order puts reasoning and evidence_quotes before score so the score conditions on the reasoning in the JSON schema sent to the model. score is int | Literal['Unknown'] (not int | None) so abstain is structurally distinct from the silent- None failure mode the old _judge_call exhibited. Four abstain-reason constants for the calibration report's cause breakdown: provider-exhausted, schema-parse, out-of-range, and the empty-string sentinel for genuine model abstain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>76e370c
-
feat(eval): Week 1 step 5 — 25-question K8s golden dataset + grounded_refusal fix
Author k8s_golden.json with 25 questions mapped to the CRAG 8-type taxonomy locked in QUESTION_PLAN.md. Distribution: 6 simple, 4 simple_w_condition, 4 comparison, 6 multi_hop, 4 false_premise (2 flavor A + 2 flavor B), 1 set. 2 questions flagged time_sensitive (k8s_005 PSA stable-since-v1.25, k8s_018 HPA autoscaling/v2). Pilot file k8s_golden_pilot.json retained unchanged as session history (tests/test_golden_schema still asserts against it). Fix grounded_refusal metric — two bugs, one semantic area. Bug 1: the metric's docstring said it checks whether the ANSWER cites no sources, but the implementation was checking whether retrieval returned zero candidates. Real agents retrieve candidates, find them irrelevant, and refuse in the answer text with no inline citations — that's the refusal shape the metric is meant to score. Fix checks for [source:...] in the answer text instead. Silent false-negative on all 5 fastapi OOS questions (q008–q010, q026–q027) which all correctly refuse but…4454894
-
feat: K8s pilot corpus — 8 pages + config entry + JSON rewrite
Land the minimal K8s corpus needed to run the 6-question pilot: - Fetch 8 pages from kubernetes.io/docs (pods, deployment, replicaset, configmap, secret, node-pressure-eviction, network-policies, pod-security-admission) via defuddle. Flat `k8s_*.md` naming mirrors the existing `fastapi_*.md` precedent. - Expand `corpora.k8s` in `configs/default.yaml`: flip `available: true`, add `golden_dataset` pointer, drop `refusal_threshold` from 0.30 placeholder to 0.02 for the pilot smoke test. Two-line inline comment preserves the 0.30 launch-intent rationale pending the full tuning sweep. - Rewrite `k8s_golden_pilot.json` `expected_sources` from path-style (`concepts/workloads/pods`) to filename stems (`k8s_pods.md`) so the exact-string match in `retrieval_precision_at_k` works. `source_pages` stays as the human-readable path anchor. - Fix three `source_snippets` that drifted from the live page text: pilot_002 (deployment rollout sentence paraphrase), pilot_003 (secre…ce7247c -
feat: add 6-question K8s golden pilot dataset
Covers all six CRAG question types: simple-fact, multi-hop, comparison, conditional, false-premise, and version-specific. Each pilot cites verbatim snippets extracted from the v1.31 Kubernetes docs snapshot (concepts/workloads/pods, controllers/ deployment, controllers/replicaset, configuration/configmap, configuration/secret, scheduling-eviction/node-pressure-eviction, services-networking/network-policies, security/pod-security-admission). Pre-backfill state: source_chunk_ids is empty on every pilot. scripts/backfill_chunk_ids.py (to be added in the K8s ingestion commit) will populate chunk IDs by substring-matching source_snippets against indexed chunks. Uses Variant B (documented-negative) for the false-premise question: NetworkPolicy mTLS points at the "What you can't do with network policies" section rather than a pure refusal. Flavor A (pure refusal) is reserved for the remaining 19 questions when the full set is drafted. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@a…3484214 -
feat: extend GoldenQuestion with source_pages and source_sections
Add source_pages and source_sections to GoldenQuestion for human-readable source attribution and pre-backfill authoring. Index-aligned with source_snippets; source_sections[i] == "" means the snippet is in page lede content above the first H2/H3 (allowed, not a missing value). New test asserts the index-alignment invariant across all four parallel source lists and allows source_chunk_ids to be empty pre-backfill or length-matched post-backfill. No behavior change for existing datasets; new fields default to empty lists. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>d5884af
-
feat: support multi-corpus golden dataset schema
GoldenQuestion gains optional source_chunk_ids, source_snippets, question_type, is_multi_hop fields (all default empty). load_golden_dataset accepts either legacy flat list or new nested format with corpus/version/snapshot_date header. Existing FastAPI dataset loads unchanged.83d6b2b
-
fix: comparison framing, mock-specific failure analysis, stale test counts
- Comparison docs now state exact config (reranker enabled for both pipelines) and cite V2 custom numbers from provider_comparison.md. Removed unsupported latency claims (runs at different times). - Report generator now gates MockProvider-specific root cause text behind provider name check. Regenerated LangChain reports. - Updated all README test counts to 169 consistently. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>a29d68d
-
fix: grounded refusal checks no-sources, reference_answer for judge, mock disclaimer
Finding 1: grounded_refusal now requires BOTH refusal language AND zero cited sources. Passing refusal text with sources = FAIL. New test: test_out_of_scope_refusal_but_has_sources. Finding 2: benchmark report now has explicit MockProvider disclaimer noting it demonstrates infrastructure, not real LLM performance. Real numbers require `make evaluate-fast` with API key. Finding 3: Golden dataset now has reference_answer field (22 populated, 5 empty for out-of-scope). Correctness judge uses reference_answer when available instead of keyword bag. Skipped when field is empty. 97 tests, lint + mypy clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>520796c -
fix: retrieval metrics use ranked sources, LLM judge wired, report complete
Finding 1 - Retrieval metrics: SearchTool now emits ranked_sources (with duplicates, preserving rank) and source_chunks (raw text) in metadata. AgentResponse carries both. Harness uses ranked_sources for P@5/R@5, not deduplicated answer sources. Finding 2 - LLM judge: Harness computes both faithfulness (with actual source_chunks) AND correctness. scripts/evaluate.py respects config.evaluation.judge_provider. Finding 3 - Report contract: Added Chunking Strategy Comparison section with instructions for comparative runs. Failure analysis includes automated root-cause detection for MockProvider runs. Added 24 evaluation-specific tests (96 total): Retrieval P@5/R@5, keyword hit rate, source presence, grounded refusal, citation accuracy, calculator check, golden dataset loading, report section verification. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>3d027cb -
feat: Day 7 — evaluation harness, metrics, report, expanded golden dataset
Evaluation: - metrics.py: 8 deterministic metrics (P@5, R@5, keyword hit rate, source presence, grounded refusal, citation accuracy, calculator check, tool call count) + 2 LLM-judge metrics (faithfulness, correctness) with JSON-structured output and logged reasoning - harness.py: async runner over golden dataset, computes all metrics per question, optional LLM judge - report.py: markdown report generator with aggregate, by-category, by-difficulty, failure analysis (3 worst), per-question detail, config snapshot Golden dataset expanded to 27 questions: - 8 easy retrieval, 8 medium retrieval, 4 hard (multi-source) - 3 calculation questions (pagination, workers, CORS) - 5 out-of-scope (Kubernetes, Kafka, GraphQL, Alembic, load balancing) - 2 extra out-of-scope beyond spec (more negative coverage is better) scripts/evaluate.py and scripts/benchmark.py wire the pipeline. docs/benchmark_report.md generated with MockProvider (template proof). Day 7 gate: benchmark report exists…c378584 -
feat: Day 4 — corpus, ingest script, first 10 golden questions
- 16 curated FastAPI tutorial markdown files (~9,400 words) with specific numbers for calculator questions (page sizes, worker formulas, token expiry, CORS max_age, etc.) - scripts/ingest.py: chunk → embed → store pipeline 207 chunks from 16 files into FAISS + BM25 hybrid store - 10 golden questions: 7 positive (incl 1 calculator), 3 out-of-scope - Pin sentence-transformers <5.0.0 for PyTorch 2.2 compat Day 4 gate PASS: Recall@5 = 1.00 on all positive questions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>a152b95