Spaces:
Sleeping
Sleeping
Commit History
-
nav: icon tabs, and a roomier bid token cap
Chat and metrics tabs become a speech bubble and a bar chart -- the analytics glyph SaaS dashboards use -- keeping the orange active state and adding aria-label/title so the words aren't lost. max_bid_tokens 3500 -> 5000, since the cap only bites on the winning speculative drafts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>ebf62dc
-
config: strip whitespace from settings, fixing the prod image search
The new /health probe caught the real cause immediately: LocalProtocolError: Illegal header value b'Bearer tvly-dev-...T17o\n' Not a bad key -- a trailing newline on the Space secret. httpx refuses to build an illegal Authorization header, so every image search failed before leaving the container, which is why a correct-looking key produced zero images on every query. str_strip_whitespace=True on the settings model fixes this for every key at once, rather than relying on secrets being pasted cleanly. Verified by starting the app with a newline-suffixed key: previously error, now {state: ok}. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>04475cf -
health: report whether image search actually works; don't eat $ as math
/health reported image_search:true for any non-empty TAVILY_API_KEY, which never validated it. That is how production looked healthy while returning zero images on every query -- the key is present in the container but the call 401s. Verified locally by simulating it: valid key -> {state: ok, detail: startup probe returned images} quoted key -> {state: error, detail: HTTP 401: Unauthorized...} no key -> {state: disabled, detail: TAVILY_API_KEY not set} A single probe runs at boot (backgrounded, so a hanging Tavily can't delay readiness) and every real search updates the same status, so the field reflects live behaviour rather than configuration. Separately: remark-math reads a pair of $ as inline-math delimiters, so "$15 million ... $85 million" rendered everything between the amounts as one stacked KaTeX expression. A $ that introduces a digit is now escaped. Real inline math from these models arrives as \(...\) or $$...$$, both already normalised before this…e4b6db2 -
citations: space before the link, and a hover preview card
The model writes the marker straight onto the preceding word, so linking it produced "Self-Defence Forcesen.wikipedia.org". A space is now inserted unless the marker already follows whitespace or an opening bracket. The annotations carry a title for each source, which was being dropped along with everything but the URL. It now rides along as the markdown link title, so no schema change was needed to get it to the client -- react-markdown hands it straight to the anchor. A custom `a` component renders it as a small rounded card above the link on hover: source title in white, host in orange, pointer-events-none so it can't swallow the click. Ordinary links (no title) render exactly as before. Where several citations share a host, the one with a title wins, since a titleless match would render a card with nothing in it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>4ec29a9
-
images: rounded uncropped tiles, and a fallback search query
Tiles get rounded corners (overflow-hidden so the image is clipped to them) and go back to object-cover: object-contain avoided cropping but left dead space around anything that wasn't 4:3. More importantly the strip was vanishing at random. Searching the answer's opening sentence is accurate but a long specific sentence sometimes matches no images at all. Now it falls back to the question's subject when the answer-derived query comes up empty -- 3/3 runs return images where it had been intermittent. Also disables rehype-highlight's language auto-detection. ASCII diagrams and timelines arrive untagged, so it guessed at them and coloured ordinary words as keywords ("site", "First", "395 CE"). Blocks the model explicitly tagged still highlight; untagged ones keep the frame and monospace without the paint. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>ac47f15 -
web: make 【domain】 citations real links
The web plugin writes citations as a bare domain in fullwidth brackets — "Narendra Modi is the Prime Minister of India【pmindia.gov.in】" — which renders as text that looks clickable and isn't. The actual URLs were there all along, in message.annotations, and llm.py read only message.content and dropped them. Now they're collected (from the streamed deltas too) and each marker is matched to the annotation whose host it belongs to, so the link lands on the cited page rather than the site's front door. A marker with no matching annotation is left exactly as it was: linking a bare domain would be a guess, and a wrong link is worse than none. Deltas still stream the raw markers; the linked text arrives with the final answer, which is what the client renders on `done`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>cd40eba -
pipeline: give the hedge the web, instead of skipping it
Web queries felt broken next to ChatGPT, and the search was not why. Measured on "bitcoin price today": 70ms bidding starts 22391ms auction resolves <- 22.3s of nothing on screen 22392ms web search starts 25264ms first token <- search + generation: 2.9s The pipeline set run_hedge=False for any query needing search, so that a stale ungrounded draft wouldn't flash before the search. Sound, but it meant the queries with the longest wait showed nothing at all, while ordinary queries had hedge tokens covering the same gap. Now the hedge runs web-grounded instead of not running: at t=0 when the regex knows, or restarted ~500ms in when the gate says so. And a web-grounded hedge is allowed to win the auction, which also skips the redundant second draft. Same query, after: first token 22391ms -> 3864ms, total 29.8s -> 15.9s. Non-web queries are unchanged. Known rough edge: when the auction picks a model other than the hint, the web hedge is discarded and t…7f8d928 -
images: search the answer, not the question; larger uncropped tiles
"who won the world cup" returned Argentina lifting the trophy under an answer about Spain. The search was built from the question, so it found whatever is most famous for those words rather than what this particular answer says. The image lookup now runs after the answer exists and searches its opening sentence, which names the entity the question only gestures at. Verified: the same query now returns Spain celebrating as 2026 champions, even though the sentence also contains "beating Argentina". The cost is that the lookup no longer overlaps the draft, so images appear about a second after the answer finishes instead of with it. That is the right trade -- a fast irrelevant picture is worse than a slightly late correct one. Tiles go from h-28/w-40 to h-44/w-60, and object-cover to object-contain: these are photos of one specific thing, and cropping to fill was cutting the subject in half. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>b494250 -
pipeline: Granite web-search gate, racing the hedge
A tiny model (ibm-granite/granite-4.1-8b) answers "does this need a web search?" at t=0, OR'd into the existing regex and bidder signals. Motivation is recall, not speed. The regex scores 100% on its own test file but 62.5% on held-out phrasing -- it misses ordinary questions like "did the fed cut rates" and "what did apple announce". The gate model scored 100% on the same held-out set (evals/bench_web_gate.py). It runs in PARALLEL with the hedge, not before it. The gate's TTFT (~450-500ms measured) is about the same as the hedge model's, so putting it on the critical path would roughly double time-to-first-token on the majority of queries, which need no search at all. Racing instead, a "yes" cancels the hedge ~500ms in rather than at auction time (2-4s), so the user sees far less doomed text before the reset. Fails open: an outage or unparseable reply returns None, which callers treat as "no opinion", so the regex and bidder flags still decide. Cost is ~$0.000007/query, tracked as …567b94e -
evals: tunable reasoning budget, so reasoning models score fairly
gpt-5-nano emits 400-1400 reasoning tokens at default effort, so the fixed 32-token cap hit finish_reason=length before any content and scored as a hard failure. With --max-tokens 600 --effort minimal it is 100% accurate -- and 1297ms p50, which is the real reason to reject it. Also accepts non-delta stream chunks; Cohere sends a whole message at once and was scoring zero on a parsing quirk rather than on merit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>4546097 -
evals: benchmark candidate models for the t=0 web-search gate
Measures TTFT and accuracy for small models answering the one yes/no question the gate asks, and scores the incumbent regex alongside them. Two datasets, because the distinction turned out to decide the result: the regex scores 100% on tests/test_web_heuristic.py, but those tests are what it was tuned against. On held-out phrasings it gets 62.5%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>a7b9c56 -
images on web-search answers, via Tavily
The OpenRouter web plugin runs search server-side and returns no image data, so image-worthy queries now make one Tavily call of their own. Gated twice over: the query must already be doing a web search, and either a bidder must set the new wants_images flag or the visual-intent regex must fire. Bidders alone proved unreliable -- they disagree with each other on identical queries and miss implicit subjects like a named office-holder or a rocket launch -- so the regex backstop mirrors what _needs_web_heuristic already does for needs_web. Search queries are stripped of conversational scaffolding first: "show me photos of the new iPhone 17" was searching for the literal words "show me photos of" and returning stock art of the phrase. Fails soft everywhere: no key, a timeout, or an empty result all just mean no images, never a degraded answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>7c53dcc
-
rich teaching answers, math rendering fix, emoji hard cap
Backend: - Teaching-style queries ("explain in detail", "how does X work", "step by step", "go deeper") discard the bid-inline answer and run a dedicated ANSWER_SYSTEM draft: the bidding context compresses answers (1 diagram) while the same model through the draft path produces the full step-by-step multi-diagram style (9 diagrams / 8 steps measured live). - _FORMATTING: skeleton exemplar for teaching answers (numbered steps, tiny diagram per step, closing full-picture diagram); diagram guidance broadened to academic/conceptual explanations and "explain further" requests. - Math rules added to the shared formatting block (tier-1 models had none): $$-delimited environments, double-backslash rows, balanced delimiters — fixes garbled matrix/aligned output. Frontend: - Markdown: wrap bare \begin{aligned|matrix|cases|...} environments in $$ (models emit them undelimited; remark-math left them as raw text), with a stash so already-delimited math is never double-wrapped. …b45aa65 -
web-need detection at t=0, shared answer formatting, friendly copy
Backend: - t=0 regex web gate (_needs_web_heuristic): phrase-based recency signals + year>=cutoff (model_knowledge_cutoff_year=2024) + _CURRENT_STATE questions (role holders, ages, net worth) with a historical-year guard. Runs before any model (~20us) and skips the speculative hedge for web queries, so a stale "I don't know" never flashes before the search. - Bidder-caught web queries (regex missed) announce "Let me search the web for you" before searching; regex-caught ones search silently. - Second-chance fallback: if a draft admits it lacks fresh info (_admits_no_fresh_info), re-answer once with live web search (stream + non-stream paths). - Shared _FORMATTING block applied to BOTH ANSWER_SYSTEM and BID_SYSTEM's speculative ---ANSWER--- (root cause of format/emoji rules being skipped when a confident bid shipped directly): answer shape, ASCII-diagram guidance, 2-3 emojis on longer answers / none on short. - tests/test_web_heuristic.py: 58 labeled cases pinning fi…f54215b -
answer formatting + UI: rich-answer format prompt, gpt-oss labels, input polish
- ANSWER_SYSTEM: prescribe a shape for longer answers (short answer, core idea, ## sections, prose/bullets, concise takeaway) with sparse emojis; greetings/simple facts stay plain. - BidArcade + MetricsDashboard: relabel the general bidder GEMMA-4 -> GPT-OSS (splash animation + metrics label; label key now matches backend "gpt-oss"). - Chat: narrower reply bubbles, taller left-aligned input pill, one-line opening tagline, centered toggles+input cluster. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>786e829 -
router: GPT-OSS 120B generalist + Llama 4 Scout verifier; greeting fix; input UI
Backend: - Generalist slot (topic "general") -> openai/gpt-oss-120b: sub-300ms median TTFT and cheaper ($0.037/$0.17 vs Llama 4 Maverick's $0.20/$0.80). Paid endpoint on both id slots so the t=0 hedge stays fast; provider_sort=latency routes to the quickest provider. - Verifier -> meta-llama/llama-4-scout, a different model from the generalist so it never grades its own drafts (keeps the quality gate independent). - VERIFY_SYSTEM: add a greetings/small-talk exception so "hello" -> a friendly reply passes at 0.9+ instead of being failed for invented "demands". Frontend (Chat input): - Circular send button with a right-arrow icon (app orange palette). - Centered toggles+input cluster; taller pill; left-aligned cursor/text. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>739705b
-
needs_web: flag specific numbered/named items (e.g. LeetCode 3499)
Qwen confabulated a LeetCode problem instead of searching. needs_web guidance now also covers a specific identified item the model can't confidently recall — a numbered problem, a specific product/version/ paper/API — 'don't invent it or say it doesn't exist, look it up'. Verified: 'leetcode 3499' -> needs_web true, real problem + citation.710d24e -
web search: tell verifier answer is web-grounded; add 'searching' stage
- verify_user gets a web_used flag (from state.needs_web): when the answer came from live web search, the verifier is told NOT to mark facts wrong/hallucinated just because they postdate its training — assume cited/web-sourced facts are current. Fixes web answers being marked unverified (2026 world cup: verify 0.0 -> 1.0). - streaming emits a 'searching' stage when web is used; UI shows '🔍 <model> SEARCHING THE WEB…'.b03e13f -
inject current date so models flag needs_web for recent events
'who won the 2026 world cup' failed to trigger web search: the winning bidder had no idea what today's date is, confidently bid needs_web=false and answered 'that hasn't happened yet' from stale training. Now every prompt is prefixed with today's date (_build_messages), and the needs_web guidance says to compare the query's date to today and not assume events 'haven't happened'. Verified: 2026 world cup -> needs_web true, web search fires, answer cites a live source.e3d533e -
bid_timeout 20->30s; narrow AI bubbles to 68%; round the input box
- bid_timeout_s 20->30: when the hint model (math specialist) is slow on a matching query, wait for its confident bid instead of escalating to the expensive frontier — was causing math queries to misroute/escalate when DeepSeek's endpoint spiked past 20s (skipped, not low-bidding) - AI answer bubbles max-w 80%->68% (~15% narrower) - query input box gets rounded corners (rounded-xl)92b2beb -
tier-1 lineup: Gemma 4 (general), Qwen3 Coder (coding), DeepSeek V4 Flash (logic/math)
- general: Gemma 4 back (google/gemma-4-26b-a4b-it, free+fallback) - logic/math: DeepSeek V4 Flash (moved from general), math specialty - creative slot + Skyfall removed; toggle creative -> reasoning (logic/math) - hint literal creative->reasoning in schema/api/Chat; labels + arcade updated - _skip_verify auto-disables (no 'creative' hint key), pipeline default -> gemma Verified: general->Gemma 1.0, coding->Qwen 0.95, reasoning->DeepSeek 1.0.f3a0f20 -
qwen: force low bids outside coding
Qwen was over-bidding on general queries despite being a coding specialist. Its specialty now explicitly instructs 0.9+ ONLY on programming/debugging/architecture and 0.4-or-lower on anything non-technical (defer to generalist even if it could answer). Verified: general queries -> qwen bids 0.4; coding -> 0.95.9284648
-
fix math rendering: strict LaTeX delimiters in frontier prompt + forgiving KaTeX
The frontier (GPT-5.6 Terra) produced garbled math — blockquoted equations (> $$), undelimited equations, and prose swallowed into unbalanced $$ blocks (rendering as stacked/run-together chars). - FRONTIER_SYSTEM now mandates: $...$ inline / $$...$$ display on its own line with blank lines, balanced delimiters, no blockquoted math, no \boxed, connective prose kept outside math with normal spacing. - rehypeKatex now throwOnError:false + strict:false so any remaining malformed LaTeX renders in place instead of cascading into garbage.8d857d1 -
skip verification entirely when the creative model wins
Creative writing has no correct answer to check, so when the creative slot (Skyfall) wins the auction, route straight to finalize (graph) and skip the verify() calls (streaming) — no verifier cost, no spurious '(unverified)' label. _skip_verify derives the creative model from the 'creative' hint mapping. Verified: haiku -> no verify stage, clean label; general query still verifies 1.0.f739b6d -
frontier: GPT-5 -> GPT-5.6 Terra
frontier_model_id default openai/gpt-5 -> openai/gpt-5.6-terra ($2.50/$15); added to _FRONTIER_SPECS. Note: plain openai/gpt-5.6 resolves to the 'sol' variant, so the explicit terra id is used. Health confirms tier2_model=openai/gpt-5.6-terra; escalation path unchanged. Baseline/savings now priced against Terra.2848bbb -
tier-1: DeepSeek V4 Flash (general), Qwen (coding), Skyfall 36B V2 (creative)
- general slot: DeepSeek V4 Flash (moved from math); Llama 4 Maverick dropped - creative slot: Skyfall 36B V2 (thedrummer/skyfall-36b-v2, paid $0.55/$0.80) replaces the math/reasoning slot; toggle 'logic/math' -> 'creative' - hint literal reasoning->creative in schema/api/Chat; labels + arcade updated - verifier: subjective exemption broadened to creative writing (poems, stories, role-play) so a competent poem isn't failed for lacking factual correctness (poem test 0.0 -> 0.7) Verified: general->DeepSeek 1.0, creative->Skyfall writes a poem.dabb994 -
general slot: Llama 3.1 70B -> Llama 4 Maverick
meta-llama/llama-4-maverick (400B MoE, paid, $0.15/$0.60) as the generalist — newer and cheaper than the 3.1 70B it replaces. Key stays 'llama'; display labels updated. Verified live: general -> Maverick wins 0.9, verify 1.0.48c2400 -
general slot: Hermes 4 70B -> Llama 3.1 70B Instruct
meta-llama/llama-3.1-70b-instruct (paid, $0.40/$0.40) as the generalist. Key hermes->llama across config, hint map, pipeline default, metrics label, bid arcade. Verified live: general query -> Llama wins 0.95, verify 1.0, 2.9s.62aaf69 -
web search: freshness-gated OpenRouter web plugin for the agents
Bidders emit a needs_web flag (does answering require current/real-time info). When the winner flagged it, the draft/hedge/frontier call enables OpenRouter's web plugin (live search injected into context, $0.004/search, 3 results). Ungrounded speculative/hedge answers are discarded for needs_web queries so the answer is web-grounded. Bidders told web results will be provided, so current-events queries bid normally and resolve at tier 1 instead of escalating. Verified live: 'most recent F1 GP' -> Hermes tier-1 with web, returns 2026-season result past training cutoff.3072e01 -
bidder answers: go long when detail is asked; raise token caps
- BID_SYSTEM answer clause + ANSWER_SYSTEM now explicitly say to write a thorough, in-depth answer when the user asks to explain/elaborate/ go into detail — several paragraphs and examples. - max_bid_tokens 1600->3500, max_answer_tokens 2000->3500 for headroom. Verified: 'explain neural networks in detail' -> 3261 chars (was ~800); 'capital of Spain' stays short.175c7ab -
simplify BID_SYSTEM per user's version (typos fixed)
Replaces the bid prompt with the user-provided shorter version. Drops the explicit typo/ambiguity bid-calibration rules, the below-0.8 omit-reason instruction, and the answer-clause typo/ambiguity/length guidance. Fixed two typos in the source ('model.Assess' -> 'model. Assess', 'only iand f' -> 'only if').23c8369 -
fix literal 'Subheading' echo in prompts; bold back to white
- prompts said to title sections with a '### Sub-heading' placeholder, which models echoed literally ('Subheading: ...'). Now: heading text names the actual topic (e.g. '### Pacing'), never the word heading. - inline **bold** rendered orange; back to white + bold per request.8c03fd5 -
general slot: Gemini 2.5 Flash Lite -> Hermes 4 70B
nousresearch/hermes-4-70b (Llama-3.1-70B hybrid reasoning, paid, $0.13/$0.40) as the generalist. Key gemini->hermes across config, hint map, pipeline default, metrics label, bid arcade. Verified live: general query -> Hermes wins 0.98, verify 1.0.0423963 -
tier-1: Gemini 2.5 Flash Lite (general), DeepSeek V4 Flash (math), Qwen3 Coder (coding)
- general: Gemini 2.5 Flash Lite (google/gemini-2.5-flash-lite, paid, $0.10/$0.40) replaces DeepSeek V4 Flash - math: DeepSeek V4 Flash moves from general to math; Nemotron dropped - coding: Qwen3 Coder unchanged - hint map, pipeline default, metrics labels, bid-arcade names updated Lighter lineup: general/math queries back to ~3-5s (was ~9s with the 550B Nemotron). Verified live: general->Gemini, reasoning->DeepSeek, both verify 1.0.eea68eb -
bidders: allow slightly more depth on explanatory answers
Replaced 'Be concise' with length-matched guidance in ANSWER_SYSTEM and the speculative-bid answer clause: short for simple factual queries, a bit more depth for explanatory ones. Verified: 'capital of Norway' stays one line; 'why is the sky blue' now gets a fuller explanation.2994af5 -
answers: point-per-line structure guidance, moderate spacing, lighter body text
- prompt: each point on its own line, explanation on the next, one blank line between points; examples set off on their own line - markdown: space-y-4 -> space-y-3, list items get my-1.5 so points separate cleanly without excess whitespace - body text stone-400 -> stone-300 (closer to white)9dddc0e -
prompt + spacing: airy answers, no wall-of-bold formatting
ANSWER_SYSTEM and FRONTIER_SYSTEM now instruct: blank lines between distinct sections, prefer short prose over dense bullet lists, don't bold every list label or stack bold headings back-to-back, reserve bold for rare key terms. Markdown section spacing space-y-2 -> space-y-4 (list-item spacing unaffected).5f6a8e5 -
revert verifier: Tencent Hy3 -> GPT-OSS 120B
Hy3 (295B reasoning) was ~5-6s per verify vs GPT-OSS's ~1.7s, and it runs on every tier-1 answer so it became the app's latency floor. Grading a draft to a small JSON verdict doesn't need a heavy reasoner. Verify stage back to ~1.7s; total query ~7.6s -> ~4.5s.40fa5b5 -
coding slot: Qwen3 Coder Flash (paid) -> Qwen3 Coder (free-first)
qwen/qwen3-coder:free primary + paid fallback restores free-first coding, so all three bidders run free-first again. Verified live: coding query -> Qwen3 Coder wins 0.95, verify 1.0.8c8accb -
tier-1 reshuffle: V4 Flash (general), Qwen3 Coder Flash (coding), Nemotron (math)
- general: DeepSeek V4 Flash (moved from math), generalist specialty - coding: Qwen3 Coder Flash — new (qwen/qwen3-coder-flash, paid, $0.195/$0.975); DeepSeek V4 Pro dropped - math/reasoning: Nemotron 3 Ultra (moved from coding), reasoning specialty - hint map, pipeline default, metrics labels, bid-arcade names updated Verified live: all three hint routes win their slot, verify 1.0.8bbe7e0 -
tier-1 lineup: DeepSeek V4 Pro (general), V4 Flash (math), Nemotron 3 Ultra (coding)
- general slot: Qwen removed; DeepSeek V4 Pro (deepseek/deepseek-v4-pro, paid, $0.435/$0.87) is the generalist. Plain deepseek/deepseek-v4 is not a callable id — the full non-Flash V4 is the Pro variant. - coding slot: Nemotron 3 Ultra (free primary + paid fallback) moves here from general; specialty rewritten for code/technical reasoning. - math slot: DeepSeek V4 Flash unchanged. - hint map, pipeline default, metrics labels, bid-arcade names updated. Verified live: general -> V4 Pro wins 0.99 verify 1.0; coding -> Nemotron wins 0.95. All three bids resolve.227a6c5 -
swap math bidder: DeepSeek chat -> DeepSeek V4 Flash
deepseek/deepseek-v4-flash (284B MoE, free primary + paid fallback, $0.09/$0.18) replaces deepseek-chat. Cheaper and now free-tier capable like the other bidders. Verified live: reasoning query -> DeepSeek V4 Flash wins 0.99, verify 1.0.c71a6f8 -
swap models: Nemotron 3 Ultra as general bidder, Tencent Hy3 as verifier
- tier-1 'general' slot: Gemma 4 -> nvidia/nemotron-3-ultra-550b-a55b (free primary + paid fallback $0.60/$3.60); key renamed gemini->nemotron across config, hint map, pipeline defaults, metrics labels, bid arcade - verifier: openai/gpt-oss-120b -> tencent/hy3 ($0.20/$0.80) - specialty rewritten: Nemotron is a strong generalist + reasoner, not a lightweight model, so it bids accordingly Verified live: general query -> Nemotron wins 0.99, Hy3 verifies 1.0.50ccdb2 -
deploy hardening: access gate, rate limiting, spend guard, env CORS
Split-deploy ready (Vercel frontend + HF backend). The API key is a server-side secret; the threat is abuse of the endpoints that spend it, so bound the worst case in layers: - access code (X-Access-Code) required on all /api/* routes via one dependency; /health stays open and reports access_required so the frontend gate auto-skips in local dev (no ACCESS_CODE set) - per-IP rate limiting (slowapi) on the query endpoints, real IP from X-Forwarded-For - in-memory daily spend guard -> 503 once DAILY_SPEND_LIMIT_USD hit; fed by every run's cost at persistence time - env-driven CORS allowlist (ALLOWED_ORIGINS) - locking the code also closes the previously-open /api/runs and /api/metrics global leaks Frontend: AccessGate screen, X-Access-Code header on all requests, 401 bounces back to the gate. README gains the split-deploy runbook + env matrix + security model. slowapi added. Verified locally: 401 without code, 429 over rate limit, 503 over spend cap, gate auto-disabled when n…98d4633
-
eval harness + env-swappable frontier model
- FRONTIER_MODEL_ID env swaps the tier-2 model (GPT-5 default; deepseek-r1 / llama-4-maverick / qwen3-235b priced) so evals avoid GPT-5 bills. - evals/: 38-item bucketed dataset (easy factual, subjective, typos, ambiguous, coding, reasoning, hard STEM) with expected routing tiers and reference notes; runner scores every answer with an independent LLM judge and reports judge score, tier-1 rate, routing accuracy, p50 latency, and cost per bucket; frontier-only baseline mode for comparison. Results JSON per run, gitignored.6d1ccea -
instruct GPT-5 to format answers in Markdown
Frontier answers arrived as plain text (no bold/headings/fences/LaTeX) so the renderer had nothing to style — tier-1 models emit markdown unprompted, GPT-5 via raw API does not. FRONTIER_SYSTEM now requests ## sections, **bold** key results, language-tagged code fences, and LaTeX math. Verified on a real escalation: headings and LaTeX present.9008432 -
hard gate: GPT-5 reserved exclusively for hard STEM/reasoning queries
Escalation (weak bids, disagreement, failed verification) is only permitted when mean bid estimated_difficulty >= 0.6 — the level only hard math/coding/reasoning queries reach. Everything else ships the best tier-1 draft; a failed verification marks the answer '(unverified)' in answered_by and the delivering event carries verified:false so the UI shows an UNVERIFIED status. Sole carve-out: all bidders erroring (no difficulty signal) keeps the frontier fallback open — that's infrastructure failure, not routing.e719504 -
speculative bid answers follow the same typo/ambiguity rules as drafts
Bid-attached answers came from BID_SYSTEM, which never received the typo/ambiguity guidance added to ANSWER_SYSTEM — so 1.0-confidence bids shipped drafts that guessed at typos or asked for clarification, failed verification, and escalated. The answer clause now carries the same rules, plus: if you'd need to clarify or refuse, you weren't 0.8 confident — bid lower and skip the answer.64bc162 -
treat ambiguity as normal, not escalation-worthy
Bidders bid on answering the most plausible interpretation; answerers answer it (noting the main alternative); the verifier accepts it as complete. min_auction_confidence 0.65->0.55 and disagreement stddev 0.22->0.30 so only drastic uncertainty escalates pre-draft — genuinely hard queries still bid 0.2-0.4 and route to GPT-5. Ambiguity probes ('tell me about mercury', 'how do i get better', 'whats the deal with interest rates') all resolve tier-1 with verify 0.9-1.0.efa41b3 -
treat obvious typos as the intended question across bid/answer/verify
'PM of infia' was tanking bid confidence and escalating to GPT-5. Bidders now bid as if the query were spelled correctly when intent is obvious, answerers answer the intended question instead of asking for clarification, and the verifier grades against the intended question. Repro queries now resolve tier-1 with verify 1.0.1622f7c