Spaces:
Running
Running
Commit History
-
fix: batch-3 adversarial review findings
Three findings from the Codex adversarial review of batch 3. [high] K8s corpus was exposed by default without a curated store CorpusConfig gains `available: bool = True`. default.yaml marks k8s as available=false, keeping it in the schema for dashboard visibility but skipping it from corpus_map at startup. AppConfig validator rejects a default_corpus pointing at an unavailable corpus. The startup log emits `corpus_skipped_unavailable` with a hint on how to re-enable. Requests for the k8s corpus now return 400 via the existing _resolve_orchestrator check, instead of silently hitting an empty HybridStore and returning a misleading refusal. Dashboard wiring: the landing-page renderer substitutes a `{{CORPUS_CONFIG_JSON}}` placeholder inside a `<script id="corpus-config" type="application/json">` block with per-server availability. The JS reads that block at load time and disables corpus toggle buttons for unavailable corpora with a "not yet available" tooltip, so the button is visible …42c7303
-
fix: address 5 review issues in dashboard frontend
1. XSS: replace all innerHTML with textContent/DOM construction where server data is interpolated (retrieval items, refusal display, iteration nodes, running-on label) 2. Wire showRetrievalRefusal: orchestrator now emits refused flag and refusal_threshold in retrieval stage done event; frontend detects and renders grounded refusal display 3. Provider toggle: now display-only, reflects server config from meta event instead of pretending to switch providers 4. PII badge: pii_redactions_count threaded through orchestrator _orchestrator_done -> route handler done event -> frontend 5. Cache HTML: index.html read once at first request, not on every / hit Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>b9c2c17 -
fix: stream stage events live, thread source_chunks, fix LangChain wrapper
- Stage events now yield immediately to the client instead of being buffered. Only the chunk is held back for output validation. - source_chunks threaded through _orchestrator_done metadata and passed to output_validator.validate(), matching /ask behavior. - LangChain AgentBenchRetriever updated for RetrievalResult wrapper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>77c4ed4 -
fix: address remaining review issues (3/5)
- Drop cosmetic reranking 'running' event — reranking completes inside tool execution, so only emit 'done' with results (issue #4) - Replace for/else with explicit response.tool_calls check, matching the pattern used by run() (issue #5) - Fix test_stream_events_ordered to filter legacy events, accommodating new stage events in the stream (issue #3) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>ffea8ea -
feat: orchestrator.run_stream emits per-stage SSE events
Yields retrieval, reranking, and llm stage events during the tool-use loop with iteration counters. Tool call events include arguments for dashboard display. Legacy sources/chunk events preserved at end. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>d0f0142
-
fix: handle max_iterations=0 in run_stream to avoid UnboundLocalError
When max_iterations=0, the tool loop never executes, leaving response unbound. Now falls back to a single toolless complete() call, matching the run() method's behavior for exhausted iterations. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>47b6a2e -
fix: eliminate redundant stream_complete() call in run_stream
The tool-use loop's final complete() already produces the synthesis when it returns with no tool_calls. Previously, run_stream discarded that response and made a second stream_complete() call — doubling the API cost for tool-using requests. Now always emit response.content from the loop's final complete() as a single chunk. This means /ask/stream no longer incrementally streams tokens for the synthesis, but it eliminates the extra LLM call and makes cost accounting exact (total_cost now covers all LLM calls). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>cafa02c -
fix: propagate LLM cost from run_stream to /ask/stream metrics
run_stream now accumulates estimated_cost_usd from complete() calls during the tool-use loop and attaches it to the done event metadata. The /ask/stream route reads this cost and passes it to metrics.record() instead of hardcoding 0.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>b8e3368 -
fix: /ask/stream parity with /ask, eliminate double LLM call
- /ask/stream now loads conversation history (session_id), records metrics, and persists Q+A to session store — matching /ask behavior - run_stream no longer makes a redundant second LLM call when no tools are needed: emits the already-received response content instead of calling stream_complete again - Landing page test count updated to 169 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>6193539
-
feat: add SQLite conversation sessions with session_id
ConversationStore backed by SQLite persists conversation history. Optional session_id on /ask loads prior turns and stores Q+A. session_id=None preserves V1 stateless behavior exactly. - MemoryConfig (enabled, db_path, max_turns) in config + default.yaml - Route handler manages session state, orchestrator accepts history param - Integration tests: stateless fallback with real store, follow-up storage, and spy test proving history reaches the orchestrator - DECISIONS.md entry, README roadmap updated - 10 new tests (138 total) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>9874438 -
feat: add streaming responses via SSE endpoint
POST /ask/stream returns Server-Sent Events with progressive chunks. Tool-use loop runs normally (non-streamed); only the final LLM synthesis is streamed for immediate user feedback. - stream_complete() added to LLMProvider with same retry/timeout handling as complete() (OpenAI streaming, Mock 3 chunks, Anthropic stub) - StreamEvent model (sources → chunk* → done) - Orchestrator.run_stream() runs tools first, then streams synthesis - /ask unchanged — /ask/stream is additive - 8 new tests: SSE format, event ordering, chunk assembly, 3-chunk assertion, streaming retry/exhaustion/timeout (128 total) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>7c40db3
-
fix: retrieval metrics use ranked sources, LLM judge wired, report complete
Finding 1 - Retrieval metrics: SearchTool now emits ranked_sources (with duplicates, preserving rank) and source_chunks (raw text) in metadata. AgentResponse carries both. Harness uses ranked_sources for P@5/R@5, not deduplicated answer sources. Finding 2 - LLM judge: Harness computes both faithfulness (with actual source_chunks) AND correctness. scripts/evaluate.py respects config.evaluation.judge_provider. Finding 3 - Report contract: Added Chunking Strategy Comparison section with instructions for comparative runs. Failure analysis includes automated root-cause detection for MockProvider runs. Added 24 evaluation-specific tests (96 total): Retrieval P@5/R@5, keyword hit rate, source presence, grounded refusal, citation accuracy, calculator check, golden dataset loading, report section verification. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>3d027cb -
fix: cross-request state bleed, metadata accuracy, empty question validation
1. Orchestrator uses local variables (req_top_k, req_strategy) instead of self._request_* — no shared state mutation on the singleton. Concurrent requests cannot overwrite each other's retrieval settings. 2. AgentResponse now carries provider/model from CompletionResponse. Routes read result.provider/result.model instead of hardcoding. 3. AskRequest.question has min_length=1 — empty strings return 422. Test updated to assert 422, not permissive 200. 72 tests, lint + mypy clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>0a9ea98 -
feat: Day 6 — serving layer with FastAPI app, routes, middleware
Serving: - app.py: application factory initializing all singletons - routes.py: POST /ask, GET /health, GET /metrics - schemas.py: AskRequest/AskResponse, HealthResponse, MetricsResponse - middleware.py: X-Request-ID, structlog, provider timeout → 504, MetricsCollector (deque-based latency percentiles) Fixes from Day 5 audit: - Orchestrator passes top_k/strategy per-request via tool kwargs instead of mutating shared SearchTool state (no cross-request bleed) - SearchTool accepts _strategy kwarg from orchestrator - Retriever.search() widened to str | None for protocol compat 7 new tests (72 total): /ask 200, /ask 422, /health, /metrics, X-Request-ID header, provider timeout → 504. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>2d8989f -
fix: wire top_k/strategy through orchestrator to SearchTool, add RAG integration test
1. SearchTool now accepts default_top_k and default_strategy at construction. Orchestrator sets these per-request before running. 2. SearchTool passes strategy to retriever.search(), closing the end-to-end path for request-level retrieval settings. 3. New integration test: real SearchTool + Retriever + HybridStore (from conftest fixtures) exercised through the orchestrator loop. 65 tests, lint + mypy clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>3fc43ec -
feat: Day 5 — agent orchestrator with tool-use loop
- Orchestrator: iterative tool-use loop (max 3 iterations) - Local message list per request, no cross-request state - Toolless final call on max-iterations fallback - Source accumulation + deduplication across iterations - Manual token usage accumulation (no Pydantic operator overloading) - AgentResponse model with answer, sources, iterations, tools_used, usage, latency_ms - SourceReference model for structured source tracking 4 new tests (64 total): AgentResponse fields, max_iterations respected, source accumulation from multiple searches, deterministic output. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>447e09c