Spaces:
Running
Running
Commit History
-
style: fix ruff lint — import sorting, line length
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>12a17f8 -
fix: rerank_score falsy-value bug and double-getattr cleanup
A rerank_score of 0.0 was falling through to r.score due to `or` truthiness. Use walrus + `is not None` for correctness while keeping getattr safety for mock objects without the field. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>d3d39b4 -
feat: enrich SearchTool metadata with scores, previews, PII count
SearchTool now returns pre_rerank_count, chunk details with reranker scores and 120-char previews, PII redaction count, and refusal threshold in metadata. Prerequisite for SSE stage events. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1d16fd9 -
feat: expose reranker scores through retrieval pipeline
CrossEncoderReranker.rerank() now returns list[tuple[Chunk, float]] instead of list[Chunk]. Retriever.search() returns RetrievalResult with pre_rerank_count metadata. SearchResult gains rerank_score field. Prerequisite for SSE stage events. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>c5573d3
-
fix: mypy errors — type annotations for pii_redactor, JSONResponse returns
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>6acda69
-
fix(security): add security to /ask/stream, wire PII redactor into SearchTool
- /ask/stream: injection detection + audit logging now runs on the streaming endpoint, matching /ask. Blocked requests return 403. - PII redactor: injected into SearchTool so retrieved chunks are redacted before entering the LLM prompt. Covers both /ask and /ask/stream since both go through the orchestrator tool loop. - Security component construction moved before tool registration in app.py so the redactor is available at SearchTool init time. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>14985f8
-
feat: add grounded refusal gate based on retrieval score threshold
SearchTool.execute() now checks the max RRF score against a configurable threshold before returning results. If no chunk scores above the threshold, the tool returns "No relevant documents found" and the LLM refuses via the system prompt. Threshold 0.02 is provisional pending an eval sweep. - refusal_threshold added to RAGConfig (default 0.0 = disabled/V1 behavior) - Gate fires before scores are dropped, includes max_score in metadata - Wired in app.py and evaluate.py - 6 new tests (103 total), lint + types clean - DECISIONS.md entry with rationale Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>c410788
-
fix: retrieval metrics use ranked sources, LLM judge wired, report complete
Finding 1 - Retrieval metrics: SearchTool now emits ranked_sources (with duplicates, preserving rank) and source_chunks (raw text) in metadata. AgentResponse carries both. Harness uses ranked_sources for P@5/R@5, not deduplicated answer sources. Finding 2 - LLM judge: Harness computes both faithfulness (with actual source_chunks) AND correctness. scripts/evaluate.py respects config.evaluation.judge_provider. Finding 3 - Report contract: Added Chunking Strategy Comparison section with instructions for comparative runs. Failure analysis includes automated root-cause detection for MockProvider runs. Added 24 evaluation-specific tests (96 total): Retrieval P@5/R@5, keyword hit rate, source presence, grounded refusal, citation accuracy, calculator check, golden dataset loading, report section verification. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>3d027cb -
feat: Day 6 — serving layer with FastAPI app, routes, middleware
Serving: - app.py: application factory initializing all singletons - routes.py: POST /ask, GET /health, GET /metrics - schemas.py: AskRequest/AskResponse, HealthResponse, MetricsResponse - middleware.py: X-Request-ID, structlog, provider timeout → 504, MetricsCollector (deque-based latency percentiles) Fixes from Day 5 audit: - Orchestrator passes top_k/strategy per-request via tool kwargs instead of mutating shared SearchTool state (no cross-request bleed) - SearchTool accepts _strategy kwarg from orchestrator - Retriever.search() widened to str | None for protocol compat 7 new tests (72 total): /ask 200, /ask 422, /health, /metrics, X-Request-ID header, provider timeout → 504. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>2d8989f -
fix: wire top_k/strategy through orchestrator to SearchTool, add RAG integration test
1. SearchTool now accepts default_top_k and default_strategy at construction. Orchestrator sets these per-request before running. 2. SearchTool passes strategy to retriever.search(), closing the end-to-end path for request-level retrieval settings. 3. New integration test: real SearchTool + Retriever + HybridStore (from conftest fixtures) exercised through the orchestrator loop. 65 tests, lint + mypy clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>3fc43ec -
fix: handle malformed top_k in SearchTool, add registry dispatch test
- SearchTool.execute() catches ValueError/TypeError on top_k coercion, falls back to default 5 instead of crashing on model-generated bad input - Added test: registry.execute() dispatches to registered tool correctly - Added test: malformed top_k="five" returns results instead of raising 40 tests, lint + mypy clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>8766b82 -
feat: Day 2 — tool system with registry, calculator, and search
- Tool ABC with definition() → ToolDefinition for provider format_tools() - ToolRegistry: dict-based register/get/execute/get_definitions - CalculatorTool: simpleeval with try/except safety wrapper - SearchTool: numbered passage format [1] (filename.md): content with source deduplication in metadata - SearchTool uses Protocol for retriever dependency (decoupled from rag/) - 15 new tests: registry CRUD, unknown tool handling, calculator valid/invalid/dangerous expressions, search formatting/empty/dedup 38 total tests, lint + mypy clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>36a9ab7