feat(scripts): add an evaluation harness for Groq model candidates

fix(data): diff citation counts against Turso itself, and write in parallel

feat(data): daily refresh job for new arXiv papers and recent citations

fix(ingest): derive point ids from the arXiv id, so overlapping runs cannot collide

fix(ingest): split date windows that exceed arXiv's 10,000-result cap

perf(data): keep up to 4 citation requests in flight under the key's rate

feat(data): append newly ingested papers to the metadata sidecar

feat(ingest): allow INGEST_DEVICE (e.g. mps) and keep fp16 to CUDA

fix(ingest): make the arXiv backfill safe to run and re-run

fix(data): ride out multi-minute Semantic Scholar error bursts

fix(data): keep keyed Semantic Scholar requests under 1 per second

fix(data): open the citation staging DB in URI mode

feat(data): refresh citation counts from Semantic Scholar

fix(ingest): write new papers to the recent shard, never the primary

chore(scripts): add the whole-site audit

feat(eval): standalone recommendation audit report

feat(eval): ranking and vector-health metrics for recommendation audits

feat(discovery): CLI for collection, baseline capture and comparison

refactor: move the live benchmark under scripts

chore: remove generated browser artifacts

feat(health): measure per-shard search latency instead of inferring it

docs(backfill): measured estimate, and the sidecar caveat that reframes it

fix(arxiv): old-style ids were collapsing onto their category

perf(backfill): batch the writes, and correct the time estimate

feat(ops): make swallowed exceptions diagnosable, and size the abstract repair

feat(paper): put comprehension on the page that was built for it

feat(ingest): add --encode-only mode

feat: arXiv ingestion pipeline

perf: local metadata sidecar + fine-grained categories

feat: Cross-encoder search reranking with MiniLM-L-6

Phase 6.5: Pipeline telemetry, search UX fixes, latency profiling

Phase 6: LightGBM reranker integration (37-feature schema, 141-tree model)

Phase 6: Add arxiv_ids export script (IDs file on HF model repo)