Spaces:
Running
Running
Commit History
-
fix(judges): four review-blocking bugs (review items 1β4 + 8)β
#1 β harness.py CompletenessJudge gate restored. Pre-supersession code gated correctness on `if q.reference_answer:`; the new per-dimension loop iterated all dims unconditionally, burning tokens on guaranteed- noisy verdicts when reference_answer was empty. Now: skip completeness when reference_answer is falsy, matching the prior contract. Test asserts the gate by mocking a judge_provider and confirming 'completeness' is absent from judge_scores when reference is "". #2 β Rubric loader was fence-blind. `## Score N` literals inside fenced code blocks in anchored examples were counted as structural level headers, producing arity-mismatch errors on rubrics that wanted to quote header-shaped strings (which the design encourages). Fix: mask fenced regions with same-length whitespace before scanning for level headers, then slice the original body at the masked-text header positions to recover level bodies with their fenced content intact. New fixture rubrics_valid_with_fenced_examples.md exβ¦9255fb5 -
test(calibration): sklearn-parity fixtures + cross-check CI testβ
Four-part discipline per the design: 1. scripts/_dev/generate_kappa_fixtures.py β committed; runs from a venv outside the project (sklearn is NOT a runtime dep but is available transitively via sentence-transformers in dev installs). 2. SKLEARN_KAPPA_FIXTURES inline constants in test file β locality preserved, type-checked, version-pinned (sklearn 1.5.2, 2026-05-04). 3. Load-bearing 'DO NOT add scikit-learn' comment. 4. Cross-check CI test (TestSklearnInputsCrossCheck) compares the inline SKLEARN_KAPPA_INPUTS against the JSON sidecar written by the generator; catches 'updated CASES list, forgot to regenerate' at CI time. Three real sklearn-parity cases now pass (imbalanced binary, three-point with one diagonal swap, weighted ordinal with linear weights). Tolerance 1e-7 accommodates sklearn's 10-decimal printed precision. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>3a2ed35 -
feat(judges): Rubric markdown loader with aggressive validationβ
Rubric loads from markdown with YAML frontmatter; validates scale, arity-matches-scale, anchored-example-per-level, frontmatter required fields. ValidationError raises with file path + field context so malformed rubrics fail at construction (Day 1) not at first judge.score call (Day 2 with API budget spent). source_hash is SHA-256 of body_markdown β immutable per file content, independent of git state. Used as ScoreResult.rubric_version so ΞΊ aggregation can group by rubric identity without cross- referencing run metadata. render_prompt(level_permutation_seed=N) deterministically permutes the ## Score sections via seeded PRNG. Seed=0 returns canonical order; this is the variance-control hook used by rubric_permute in Phase 3. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>7b72b2c