Spaces:
Sleeping
Sleeping
Commit History
-
feat(goldens): add source_snippets to 8 FastAPI calibration items
Hand-snippeted verbatim spans from data/tech_docs/ for the 8 FastAPI items in calibration_v1. OOS items get source_snippets:[] (no source to ground against). Scope discipline: only the 8 calibration items, not the full 27-item FastAPI golden — the remaining 19 backfill in v1.1. Required by GroundednessJudge (reference-based on source_snippets). K8s items already had this field from the multi-corpus refactor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>a48afb9 -
feat(calibration): 30-item stratified calibration_v1 sample
Stratified across FastAPI (categorized) + K8s (CRAG question_types) per the design doc's sampling table. 26 items from explicit strata + 4 spare slots from K8s simple_w_condition / multi_hop / comparison / false_premise (highest-variance R@5 strata in pre-judge runs). Sample seed 20260504 (date-derived) so the sampling is reproducible. The spare pool was widened from {simple_w_condition, multi_hop} to also include {comparison, false_premise} because the K8s golden set holds only 4 simple_w_condition and 6 multi_hop items, of which the explicit targets already consumed 7, leaving 3 in the original pool — one short of the 4 spares the design's 30-item total requires. system_config_git_sha pins the commit producing the sample; the soon- to-be-generated system_outputs file references this SHA so any drift in the pipeline between sample generation and output generation is detectable in the calibration report. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>8ef480a
-
feat(eval): Week 1 step 5 — 25-question K8s golden dataset + grounded_refusal fix
Author k8s_golden.json with 25 questions mapped to the CRAG 8-type taxonomy locked in QUESTION_PLAN.md. Distribution: 6 simple, 4 simple_w_condition, 4 comparison, 6 multi_hop, 4 false_premise (2 flavor A + 2 flavor B), 1 set. 2 questions flagged time_sensitive (k8s_005 PSA stable-since-v1.25, k8s_018 HPA autoscaling/v2). Pilot file k8s_golden_pilot.json retained unchanged as session history (tests/test_golden_schema still asserts against it). Fix grounded_refusal metric — two bugs, one semantic area. Bug 1: the metric's docstring said it checks whether the ANSWER cites no sources, but the implementation was checking whether retrieval returned zero candidates. Real agents retrieve candidates, find them irrelevant, and refuse in the answer text with no inline citations — that's the refusal shape the metric is meant to score. Fix checks for [source:...] in the answer text instead. Silent false-negative on all 5 fastapi OOS questions (q008–q010, q026–q027) which all correctly refuse but…4454894
-
feat: K8s pilot corpus — 8 pages + config entry + JSON rewrite
Land the minimal K8s corpus needed to run the 6-question pilot: - Fetch 8 pages from kubernetes.io/docs (pods, deployment, replicaset, configmap, secret, node-pressure-eviction, network-policies, pod-security-admission) via defuddle. Flat `k8s_*.md` naming mirrors the existing `fastapi_*.md` precedent. - Expand `corpora.k8s` in `configs/default.yaml`: flip `available: true`, add `golden_dataset` pointer, drop `refusal_threshold` from 0.30 placeholder to 0.02 for the pilot smoke test. Two-line inline comment preserves the 0.30 launch-intent rationale pending the full tuning sweep. - Rewrite `k8s_golden_pilot.json` `expected_sources` from path-style (`concepts/workloads/pods`) to filename stems (`k8s_pods.md`) so the exact-string match in `retrieval_precision_at_k` works. `source_pages` stays as the human-readable path anchor. - Fix three `source_snippets` that drifted from the live page text: pilot_002 (deployment rollout sentence paraphrase), pilot_003 (secre…ce7247c -
feat: add 6-question K8s golden pilot dataset
Covers all six CRAG question types: simple-fact, multi-hop, comparison, conditional, false-premise, and version-specific. Each pilot cites verbatim snippets extracted from the v1.31 Kubernetes docs snapshot (concepts/workloads/pods, controllers/ deployment, controllers/replicaset, configuration/configmap, configuration/secret, scheduling-eviction/node-pressure-eviction, services-networking/network-policies, security/pod-security-admission). Pre-backfill state: source_chunk_ids is empty on every pilot. scripts/backfill_chunk_ids.py (to be added in the K8s ingestion commit) will populate chunk IDs by substring-matching source_snippets against indexed chunks. Uses Variant B (documented-negative) for the false-premise question: NetworkPolicy mTLS points at the "What you can't do with network policies" section rather than a pure refusal. Flavor A (pure refusal) is reserved for the remaining 19 questions when the full set is drafted. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@a…3484214
-
fix: grounded refusal checks no-sources, reference_answer for judge, mock disclaimer
Finding 1: grounded_refusal now requires BOTH refusal language AND zero cited sources. Passing refusal text with sources = FAIL. New test: test_out_of_scope_refusal_but_has_sources. Finding 2: benchmark report now has explicit MockProvider disclaimer noting it demonstrates infrastructure, not real LLM performance. Real numbers require `make evaluate-fast` with API key. Finding 3: Golden dataset now has reference_answer field (22 populated, 5 empty for out-of-scope). Correctness judge uses reference_answer when available instead of keyword bag. Skipped when field is empty. 97 tests, lint + mypy clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>520796c -
feat: Day 7 — evaluation harness, metrics, report, expanded golden dataset
Evaluation: - metrics.py: 8 deterministic metrics (P@5, R@5, keyword hit rate, source presence, grounded refusal, citation accuracy, calculator check, tool call count) + 2 LLM-judge metrics (faithfulness, correctness) with JSON-structured output and logged reasoning - harness.py: async runner over golden dataset, computes all metrics per question, optional LLM judge - report.py: markdown report generator with aggregate, by-category, by-difficulty, failure analysis (3 worst), per-question detail, config snapshot Golden dataset expanded to 27 questions: - 8 easy retrieval, 8 medium retrieval, 4 hard (multi-source) - 3 calculation questions (pagination, workers, CORS) - 5 out-of-scope (Kubernetes, Kafka, GraphQL, Alembic, load balancing) - 2 extra out-of-scope beyond spec (more negative coverage is better) scripts/evaluate.py and scripts/benchmark.py wire the pipeline. docs/benchmark_report.md generated with MockProvider (template proof). Day 7 gate: benchmark report exists…c378584 -
feat: Day 4 — corpus, ingest script, first 10 golden questions
- 16 curated FastAPI tutorial markdown files (~9,400 words) with specific numbers for calculator questions (page sizes, worker formulas, token expiry, CORS max_age, etc.) - scripts/ingest.py: chunk → embed → store pipeline 207 chunks from 16 files into FAISS + BM25 hybrid store - 10 golden questions: 7 positive (incl 1 calculator), 3 out-of-scope - Pin sentence-transformers <5.0.0 for PyTorch 2.2 compat Day 4 gate PASS: Recall@5 = 1.00 on all positive questions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>a152b95