Spaces:
Sleeping
Sleeping
Commit History
-
calibrate(jury): v1.1+v1.1.1 β fix weighting bugs; recency-position paraphrase clauseβ
Three nested findings on the judge layer's calibration set, all discoverable from `docs/_generated/kappa_table.md` joined to the per- member sidecar at `results/...jury_kappa_weighted_*_members.jsonl`. Full mechanism + v1.2 escalations in DECISIONS.md "v1.1 jury rescue" and "Plan 3A" entries. 1. v1.1 jury rescue (ΞΊ 0.014 β 0.416 on completeness, no API spend). Two compounding v1 bugs forced equal weights and let gpt-4o-mini's downward bias win every disputed cell: `_load_weights_from_baseline` was a stub returning 1.0 from a Haiku-only source, and `jury.py`'s missing-key fallback returned 1.0 for any member not in the source. v1.1 replaces the stub with `_compute_kappa_weights` (real per-judge ΞΊ vs gold, hard-error on asymmetric coverage, negative-ΞΊ clipped to 0) and converts the missing-weight fallback to a `ValueError`. Re- aggregation of the existing 164 sidecar rows with corrected weights recovers the Haiku-baseline ΞΊ exactly. Pre-committed Outcome 2 β¦ab0e054
-
fix(types): four mypy errors blocking CIβ
I ran ruff locally after each fix batch but skipped mypy (the project's make lint runs all three; I only invoked the ruff portion). CI caught what I should have. #1 harness.py:200 β _JUDGE_CLASS_BY_DIMENSION was inferred as dict[str, type[Judge]], so the dispatch site looked like an attempted instantiation of an abstract class. Annotated with the explicit union of concrete subclasses so mypy sees instantiability. #2 report.py:64 β abstain filter doesn't narrow ScoreResult.score from `int | Literal["Unknown"]` to int for mypy. Cast through float() with a targeted `# type: ignore[arg-type]` since the runtime guarantee (abstained=False β score is int) isn't expressible in the type system without overloads. #3 rubric_permute.py:75 β local `score: int | str = "Unknown"` should be `int | Literal["Unknown"]` to match ScoreResult.score's signature. #4 citation_faithfulness.py:169 β same fix: aggregate_score typed as `int | Literal["Unknown"]` instead of `int | str`. mypy now clean; 523 teβ¦02b8717 -
fix(judges): four review-blocking bugs (review items 1β4 + 8)β
#1 β harness.py CompletenessJudge gate restored. Pre-supersession code gated correctness on `if q.reference_answer:`; the new per-dimension loop iterated all dims unconditionally, burning tokens on guaranteed- noisy verdicts when reference_answer was empty. Now: skip completeness when reference_answer is falsy, matching the prior contract. Test asserts the gate by mocking a judge_provider and confirming 'completeness' is absent from judge_scores when reference is "". #2 β Rubric loader was fence-blind. `## Score N` literals inside fenced code blocks in anchored examples were counted as structural level headers, producing arity-mismatch errors on rubrics that wanted to quote header-shaped strings (which the design encourages). Fix: mask fenced regions with same-length whitespace before scanning for level headers, then slice the original body at the masked-text header positions to recover level bodies with their fenced content intact. New fixture rubrics_valid_with_fenced_examples.md exβ¦9255fb5 -
feat(scripts): run_calibration.py orchestrator for Steps A/C/Dβ
Three subcommands, all sharing concurrency-resolution + structured logging: generate-outputs β Step A: orchestrator against 30 calibration items, frozen config, writes results/calibration_v1_system_outputs.json run-judges β Step C: takes --row-config=<path>, scores frozen outputs with that row's judges, writes results/calibration_v1_judge_<label>.json build-table β Step D: invokes generate_kappa_table; --strict raises on missing predictions/labels Resolved concurrency value logged at every run so artifacts capture which concurrency was used. Default 5; CLI overrides config-field fallback overrides hardcoded default. Step B (hand-labeling) is manual β done in a Jupyter notebook, not orchestrated by this script. Also folded in lint fixes for the Phase 1-3 modules to satisfy ruff E402 (test imports moved to top of test_judges.py) and E501 (jury.py reasoning β¦4fa7c61 -
feat(variance): PermutedJudge + Jury β N permutations and multi-judge aggregatorβ
PermutedJudge wraps a Judge and runs N permutations with different prompt_seeds. Discretization: binary thresholds at 0.5 with ties β 0 (conservative); three-point rounds to nearest with ties β lower level. Any abstain in any permutation propagates to Unknown β the variance signal that permutation is designed to surface should not be averaged away. Per-permutation ScoreResults written to sidecar JSONL (append- mode so multiple score() calls across items accumulate). Jury aggregates a list of Judge instances via asyncio.gather with return_exceptions=False so non-retryable exceptions cancel sibling tasks immediately (failing fast on caller bugs). Per-member ScoreResults written to sidecar β successes AND failure-as-abstains β so the calibration report can compute per-judge ΞΊ even when the aggregate row drops to abstain via the quorum gate. Strict quorum default (quorum = len(judges)) at v1's 2-judge jury means any member abstain β jury abstain. Tolerant defaults at N=2 are silent singlβ¦c038a7d