Spaces:
Running
Running
Commit History
-
docs(backfill): measured estimate, and the sidecar caveat that reframes it
siddhm11 +1: Claude Opus 5Repairing Turso does not reach production by itself. turso_svc.fetch_metadata_batch reads LRU -> local sidecar -> Turso, and the sidecar carries abstract_preview for all 1,799,348 rows, so Turso is essentially never consulted for metadata in production. It is read-only and baked into the image from a pinned HF dataset revision. So the backfill is step 1 of 4: rebuild the sidecar, upload it, repoint METADATA_SIDECAR_URL, redeploy. Worth writing down because the obvious mental model — fix the database, the app shows the fix — is wrong here, and acting on it means a long job with no visible result. The estimate is now measured rather than modelled. Three earlier figures were wrong: 13.6h counted only the arXiv pause; 22.6h added a guessed write cost but kept the inflated 90.6% truncation count; 10h fixed the count but kept the guess. Real batches give ~7.75s per 200 rows, so ~15.5 hours for the remaining 1,436,005. Validated before launching: 400/400 fetched, zero missing (the old-styl…4f06b54
-
fix(arxiv): old-style ids were collapsing onto their category
siddhm11 +1: Claude Opus 5_normalise_id matched [^\s/v]+, which stops at a slash. Old-style arXiv ids carry a category prefix — math/0309136v1, hep-ph/0512038v2, acc-phys/9502001 — and 115,604 papers in the corpus (6.4%) use that form. Every one normalised to just its category: math/0309136v1 and math/0511124v2 both became "math". Two silent consequences. A lookup by the real id missed, so the arXiv metadata fallback returned nothing for 6.4% of the corpus. And distinct papers collided on one key, so one paper's metadata could be returned for another — the same defect class this codebase already documents for Qdrant point ids. The character class also treated a literal 'v' as a terminator. Rewritten procedurally: strip a known prefix, strip a trailing version. Two independent operations, and one regex doing both is what hid this. None of the existing parametrised cases covered an old-style id, which is why it survived. Six now do, plus a collision test and a §3.9 string-type test. Found while writing the ba…da80710 -
perf(backfill): batch the writes, and correct the time estimate
siddhm11 +1: Claude Opus 5The write path issued one UPDATE per row. At ~150ms per Turso round trip that is not a detail: a full sweep of 1.63M rows would have spent ~68 hours on HTTP alone, dwarfing the 13.6 hours of arXiv rate limiting the estimate was actually based on. Turso's pipeline API takes a batch of statements in one request, so a batch is what it gets now — 16,303 write round trips instead of 1,630,273. The estimate was wrong for the same reason: it counted only the arXiv leg. Counting both puts a full sweep at ~22.6 hours, not 13.6. Better to have the real number before committing to the run. A test fails if a per-row await reappears inside the write loop, checked with ast rather than a substring so a reformatting cannot slip past it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>1e47cba -
feat(ops): make swallowed exceptions diagnosable, and size the abstract repair
siddhm11 +1: Claude Opus 5Two things needing attention, both measured rather than assumed. ERRORS grep -rn "traceback|exc_info" app/ returned ZERO hits across the whole application, against 95 `except Exception` handlers. Every production failure was a single line with no traceback and no line number. Worse, the handler that reports feed failures sits at the bottom of a 473-line try covering clustering, quota, retrieval, metadata, reranking, MMR and labelling — so "multi-interest preprocessing failed: KeyError('x')" narrowed the fault to "somewhere in the feed" while the user silently dropped to Tier 2. Swallowing is the right CHOICE there: a feed that degrades beats one that 500s, and those paths are deliberate. Swallowing SILENTLY was not. app/errors.py adds report(), which prints the exception type, the failing file and line, a traceback trimmed to OUR frames — an untrimmed trace through httpx and asyncio buries the one frame that matters — and any chained cause, which is where the answer usually is. Sti…43e5407