Spaces:
Sleeping
Sleeping
Commit History
-
calibrate(jury): v1.1+v1.1.1 β fix weighting bugs; recency-position paraphrase clauseβ
Three nested findings on the judge layer's calibration set, all discoverable from `docs/_generated/kappa_table.md` joined to the per- member sidecar at `results/...jury_kappa_weighted_*_members.jsonl`. Full mechanism + v1.2 escalations in DECISIONS.md "v1.1 jury rescue" and "Plan 3A" entries. 1. v1.1 jury rescue (ΞΊ 0.014 β 0.416 on completeness, no API spend). Two compounding v1 bugs forced equal weights and let gpt-4o-mini's downward bias win every disputed cell: `_load_weights_from_baseline` was a stub returning 1.0 from a Haiku-only source, and `jury.py`'s missing-key fallback returned 1.0 for any member not in the source. v1.1 replaces the stub with `_compute_kappa_weights` (real per-judge ΞΊ vs gold, hard-error on asymmetric coverage, negative-ΞΊ clipped to 0) and converts the missing-weight fallback to a `ValueError`. Re- aggregation of the existing 164 sidecar rows with corrected weights recovers the Haiku-baseline ΞΊ exactly. Pre-committed Outcome 2 β¦ab0e054
-
fix(judges,calibration,harness): three Codex adversarial-review findingsβ
Codex Fix #1 β Single-row ablation options now actually take effect. Pre-fix, scripts/run_calibration.py never read row['options'], so baseline_no_cot, baseline_no_anchors, and baseline_no_abstain produced identical ΞΊ to baseline. The ΞΊ ablation table would have been methodologically invalid. Three knobs threaded through: - use_cot: Judge ABC accepts use_cot=True default; concrete judges emit a JSON schema clause via _json_schema_clause() helper that drops 'reasoning' and 'evidence_quotes' when False. The helper parser tolerates missing 'reasoning' key (data.get default '') so no_cot responses don't fail validation. - use_anchors: implemented via Rubric.strip_anchors() returning a copy with ### Example sections removed. body_markdown changes β source_hash naturally diverges β ScoreResult.rubric_version bucketing is automatic. - abstain_allowed: Judge ABC accepts abstain_allowed_override=None; effective_abstain_allowed property returns override when set, else rubric.aβ¦226b6f4 -
fix(judges,calibration): five review follow-ups (items 5, 6, 7, 9, 10)β
#5 β Gwet's AC2 weighted variant now raises NotImplementedError. The unweighted AC1 formula is correct and tested against three hand- computed cases. The weighted variant has multiple inconsistent literature definitions (Gwet 2008 vs 2014) and no sklearn analogue to cross-check, so shipping a plausible-looking weighted formula without a fixture is a methodology hazard. Gate it explicitly until v1.1 pins both formula choice and fixture. test_weighted_variant_ raises_not_implemented covers both 'linear' and 'quadratic' kwargs. #6 β citation_faithfulness empty-claim handling. When the answer starts with a [source:] citation (no prior content), the extractor returns an empty claim string. Previously the judge built a prompt with empty content and burned an API call asking the LLM to evaluate emptiness. Now: vacuously faithful (score=1, no API call), with a synthetic ScoreResult so per-pair detail still appears in evidence_quotes. test_leading_citation_empty_claim_vacuously_ faithful asserβ¦71ec5e8 -
fix(judges): four review-blocking bugs (review items 1β4 + 8)β
#1 β harness.py CompletenessJudge gate restored. Pre-supersession code gated correctness on `if q.reference_answer:`; the new per-dimension loop iterated all dims unconditionally, burning tokens on guaranteed- noisy verdicts when reference_answer was empty. Now: skip completeness when reference_answer is falsy, matching the prior contract. Test asserts the gate by mocking a judge_provider and confirming 'completeness' is absent from judge_scores when reference is "". #2 β Rubric loader was fence-blind. `## Score N` literals inside fenced code blocks in anchored examples were counted as structural level headers, producing arity-mismatch errors on rubrics that wanted to quote header-shaped strings (which the design encourages). Fix: mask fenced regions with same-length whitespace before scanning for level headers, then slice the original body at the masked-text header positions to recover level bodies with their fenced content intact. New fixture rubrics_valid_with_fenced_examples.md exβ¦9255fb5 -
feat(config): add evaluation.judge_dimensions fieldβ
Default ['groundedness', 'relevance', 'completeness'] β the v1 dimensions that have rubrics + judges + calibration coverage. citation_faithfulness is opt-in v1 (default-on v1.1) so the citation deterministic-vs-LLM head-to-head is decoupled from the harness migration. judge_provider field unchanged β preserves the YAML knob across configs/{default,production,anthropic,selfhosted_local, selfhosted_modal}.yaml. Zero user-facing config migration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>12cb8b7 -
feat(calibration): generate_kappa_table with strict/warn modesβ
Joins predictions β labels by (item_id, dimension, system_output_hash). Hash mismatch ALWAYS raises with first-item expected/actual hashes plus full mismatched-id list β applies in both modes (never warned). Missing predictions/labels warn-and-exclude by default; --strict raises (the final-artifact path; make calibrate uses it). Pairwise abstain exclusion in ΞΊ; per-dimension cause breakdown (schema_parse / out_of_range / provider_exhausted / genuine) via the abstain-reason constants from judges/base.py. Abstain-rate flag fires on STRICTLY greater than 20%; 6/30 (=20%) does not fire, 7/30 does β boundary tested explicitly. ΞΊ undefined β 'β' with footnote (insufficient variance, N<3 agreement-eligible items remaining, or all labels+predictions in a single category β the last condition was load-bearing for the 'all-ones degenerate' test case). Skips '_members.jsonl' sidecar files in the predictions glob β they hold per-permutation / per-jury-member detail, not aggregate predictions. Cβ¦1d47106 -
feat(scripts): run_calibration.py orchestrator for Steps A/C/Dβ
Three subcommands, all sharing concurrency-resolution + structured logging: generate-outputs β Step A: orchestrator against 30 calibration items, frozen config, writes results/calibration_v1_system_outputs.json run-judges β Step C: takes --row-config=<path>, scores frozen outputs with that row's judges, writes results/calibration_v1_judge_<label>.json build-table β Step D: invokes generate_kappa_table; --strict raises on missing predictions/labels Resolved concurrency value logged at every run so artifacts capture which concurrency was used. Default 5; CLI overrides config-field fallback overrides hardcoded default. Step B (hand-labeling) is manual β done in a Jupyter notebook, not orchestrated by this script. Also folded in lint fixes for the Phase 1-3 modules to satisfy ruff E402 (test imports moved to top of test_judges.py) and E501 (jury.py reasoning β¦4fa7c61 -
test(calibration): sklearn-parity fixtures + cross-check CI testβ
Four-part discipline per the design: 1. scripts/_dev/generate_kappa_fixtures.py β committed; runs from a venv outside the project (sklearn is NOT a runtime dep but is available transitively via sentence-transformers in dev installs). 2. SKLEARN_KAPPA_FIXTURES inline constants in test file β locality preserved, type-checked, version-pinned (sklearn 1.5.2, 2026-05-04). 3. Load-bearing 'DO NOT add scikit-learn' comment. 4. Cross-check CI test (TestSklearnInputsCrossCheck) compares the inline SKLEARN_KAPPA_INPUTS against the JSON sidecar written by the generator; catches 'updated CASES list, forgot to regenerate' at CI time. Three real sklearn-parity cases now pass (imbalanced binary, three-point with one diagonal swap, weighted ordinal with linear weights). Tolerance 1e-7 accommodates sklearn's 10-decimal printed precision. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>3a2ed35 -
feat(calibration): hand-rolled cohen_kappa, gwets_ac2, bootstrap_ciβ
Hand-rolled (not sklearn) per the design's 'built from primitives' discipline. Cohen's ΞΊ: (P_o - P_e) / (1 - P_e), supports unweighted, linear, and quadratic weight matrices for ordinal scales. Gwet's AC1/AC2: chance term = (1/(q-1)) Ξ£ pi_k(1-pi_k) on mean marginals, more robust to skewed distributions than Cohen's ΞΊ. Bootstrap CI: seed=42 for reproducibility, drops degenerate resamples (all-one- label) silently rather than raising. Three hand-computed test cases per metric (perfect agreement ΞΊ=1, complete disagreement ΞΊ=-1, chance agreement ΞΊ=0) include worked- out arithmetic in comments so a reader can verify formula correctness without running the test. sklearn-parity scaffold included with placeholder fixtures and a skip until scripts/_dev/generate_kappa_fixtures.py runs in a venv with sklearn==1.5.2; the cross-check test is also gated on the JSON sidecar existing. Phase 4 Task 4.2 lands the generator. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>6ef2e0e -
feat(variance): PermutedJudge + Jury β N permutations and multi-judge aggregatorβ
PermutedJudge wraps a Judge and runs N permutations with different prompt_seeds. Discretization: binary thresholds at 0.5 with ties β 0 (conservative); three-point rounds to nearest with ties β lower level. Any abstain in any permutation propagates to Unknown β the variance signal that permutation is designed to surface should not be averaged away. Per-permutation ScoreResults written to sidecar JSONL (append- mode so multiple score() calls across items accumulate). Jury aggregates a list of Judge instances via asyncio.gather with return_exceptions=False so non-retryable exceptions cancel sibling tasks immediately (failing fast on caller bugs). Per-member ScoreResults written to sidecar β successes AND failure-as-abstains β so the calibration report can compute per-judge ΞΊ even when the aggregate row drops to abstain via the quorum gate. Strict quorum default (quorum = len(judges)) at v1's 2-judge jury means any member abstain β jury abstain. Tolerant defaults at N=2 are silent singlβ¦c038a7d -
feat(judges): CitationFaithfulnessJudge with all-or-nothing aggregationβ
Per-(claim, citation) binary judge that aggregates to one item-level ScoreResult via all-or-nothing β any unfaithful citation β score=0. Per-pair detail preserved in evidence_quotes. Opt-in for v1 (judge_dimensions default excludes it); default-on in v1.1 once the citation-deterministic-vs-LLM head-to-head section of the writeup validates the gain over the existing regex-based citation_accuracy. Any sub-call abstain propagates to the aggregate (consistent with the strict-quorum / any-abstain principles in jury and rubric_permute). _extract_claims_with_citations strips prior [source:] tags from the left context before searching for sentence boundaries, so multi- citation answers yield clean claim strings rather than carrying prior citation tags into downstream claims. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>04d9ea0 -
feat(judges): CompletenessJudge + three-point reference-based rubricβ
Three-point rubric (none / partial / full) scored against the gold reference_answer. Coverage-of-facts framing: score only on what fraction of the reference's points are present, not on additional correct facts. Two anchored examples per level. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>80be2d8 -
feat(judges): RelevanceJudge + anchored three-point rubricβ
Three-point rubric, reference-free (judge sees only question + answer, not gold snippets or reference answer). Score the topic-match, not the truth-value. Two anchored examples per level (six total) covering wrong-topic, incomplete-engagement-refusal, adjacent-but-off-target, sibling- question-answer, and on-target single-fact + comparison cases. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>b170eb6 -
feat(judges): GroundednessJudge + anchored binary rubricβ
Binary rubric, reference-based on item.source_snippets. Strict- conjunction definition: any unsupported claim fails the rubric. Two anchored examples per score level with thinking-trace explanations (per Singla et al. 2025 β through-the-judge's-eyes). Per the design doc's sequencing tactic, groundedness is authored first and dry-fitted before relevance and completeness β this converts rubric authoring from three parallel risky tasks to one risky task plus two near-mechanical replications. system_output_hash is SHA-256 of (item.id, answer, sorted(sources)) joined by NUL; the calibration report uses this as the agreement- eligibility key against labels. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>30a5e0c -
feat(judges): _call_judge_with_retry helper with strict-reprompt + abstainβ
Module-level helper (not a Judge method) called by every concrete judge. One retry with the strict-formatting reprompt suffix on schema-parse / score-out-of-range; abstain with structured-prefix reason on persistent failure or provider exhaustion. Caller bugs (401, 400, anything not in the retryable taxonomy) propagate. The first-attempt-failure WARN log fires regardless of whether the retry succeeds β the silent recover-on-retry case is the most analytically interesting one for offline analysis (tells you the reprompt is doing work, not just costing 2x tokens). Test uses structlog.testing.capture_logs (not pytest caplog) since the project does not configure structlog to route through stdlib logging β capture_logs is the structlog-native test seam. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>ff78845 -
feat(judges): MockJudge with LookupError on missing keysβ
MockJudge raises LookupError (not a default) on missing item.id keys, so test fixtures are self-checking against rename drift. A separate fixture-validation test in Phase 8 walks item.id across all goldens and asserts coverage; the LookupError is the second layer of defense. __init__.py re-exports the public surface for ergonomic imports (from agent_bench.evaluation.judges import Judge, ScoreResult, ...). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>aa70e89 -
feat(judges): Judge ABC with judge_id derived from model + dimensionβ
Judge is abstract β concrete subclasses (groundedness, relevance, completeness, citation_faithfulness) land in Phase 2 as thin ~30-line classes per the no-shared-base-method discipline. judge_id format: '{model_id}_{rubric.dimension}', e.g. 'claude-haiku-4-5_groundedness'. The format is load-bearing for the calibration report's per-judge ΞΊ breakdown β the report groups by judge_id when computing per-judge agreement against the human labels. Cross-package types (GoldenQuestion, AgentResponse, LLMProvider) are imported under TYPE_CHECKING to keep judges/base.py loadable before harness.py is migrated in Phase 8 (avoiding a circular import once harness imports back from judges). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>2192305 -
feat(judges): Rubric markdown loader with aggressive validationβ
Rubric loads from markdown with YAML frontmatter; validates scale, arity-matches-scale, anchored-example-per-level, frontmatter required fields. ValidationError raises with file path + field context so malformed rubrics fail at construction (Day 1) not at first judge.score call (Day 2 with API budget spent). source_hash is SHA-256 of body_markdown β immutable per file content, independent of git state. Used as ScoreResult.rubric_version so ΞΊ aggregation can group by rubric identity without cross- referencing run metadata. render_prompt(level_permutation_seed=N) deterministically permutes the ## Score sections via seeded PRNG. Seed=0 returns canonical order; this is the variance-control hook used by rubric_permute in Phase 3. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>7b72b2c -
feat(judges): ScoreResult + abstain-reason constantsβ
ScoreResult is the per-call record; field order puts reasoning and evidence_quotes before score so the score conditions on the reasoning in the JSON schema sent to the model. score is int | Literal['Unknown'] (not int | None) so abstain is structurally distinct from the silent- None failure mode the old _judge_call exhibited. Four abstain-reason constants for the calibration report's cause breakdown: provider-exhausted, schema-parse, out-of-range, and the empty-string sentinel for genuine model abstain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>76e370c -
test: scaffold tests/evaluation/ directory for judge-layer testsβ
Phase 1+ judge tests will land under tests/evaluation/ matching the new agent_bench/evaluation/judges,rubrics,variance,calibration/ subpackages. Pattern precedent: tests/test_langchain_baseline/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>f94cea7