Spaces:
Running
Running
Commit History
Yesterday
-
fix(groq): keep cached explanations, let a key take concurrent requests, check routed rewrites like direct ones Running
siddhm11 +1: Claude Opus 5.5Review of the key-rotation commit before release found four defects: - The explanation cache key gained the title. With failover off, every cached explanation would miss once after a deploy (a Groq call each), and feed cards, which read the old key in card_data, would never show one written after it. The title is out again: for the head of the chain the key is the old key, and cards also read a fallback model's entry. - A key served one request at a time. With the one key there is, every Groq call queued: a 2 s rewrite waited behind an 8 s explanation and came back unrewritten. Groq limits requests and tokens per minute, not connections; only a recovering scope is now held to one probe. - The routed rewrite skipped _reads_like_a_reply, so a rewrite that answers the reader would have been searched. - It also refused rewrites over 8 words that the direct path accepts, spending a GPT-OSS call on them. Live check with the local key (scripts/check_groq_routes.py --probe …60e843f -
feat(groq): rotate credentials before falling back to another model
siddhm11(cherry picked from commit a40fa57bed16db9744167f921373506f1efb3fb6)a84b508 -
fix(rate-limit): limit the 3D map's API, whose search spends Groq quota
siddhm11 +1: Claude Opus 5.5/api/space/* had no rate limit and, with SPACE_SERVICE_TOKEN unset on the Space, answers anyone. /api/space/search runs the whole search, Groq rewrite included. It now gets /search's 30 a minute per address; the cheap vector reads (locate, similarity, neighbours) get 120. The map calls from its own server and forwards no visitor address, so its readers share whichever address that server calls from: hence the looser limit on the reads. A per-address limit caps bursts, not a day's Groq use; setting the token is what keeps strangers out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>456f47b -
feat(trending): cards open on the paper's figure, and /trending puts papers first
siddhm11 +1: Claude Opus 5.5Trending cards had no picture and /trending led with 16 topic boxes, so a logged-out visitor scrolled about 3,300 px before reaching a paper. - Pictures come from our own figure reader, never Hugging Face: of the week's 312 papers only 112 had HF media (73 of them MP4) and HF's thumbnail is the PDF's first page, while our reader had a figure for 179 of 187 papers read, served from our origin with its credit. - card_data.add_figures sets each card's figure from saved sources only; first_figure is shared with the feed card. - trending_shelf.warm_figures queues the week's unread papers for figure_prewarm on every refresh, most upvoted first (at most 200), and the moving topics' top papers after each topics refresh. - The card: a 16:10 plate that shimmers until the figure paints, then fades it in (app.js); no figure yet or a failed one shows the paper's name ("WeVisOne-VL") or its upvotes instead of an empty box. It names the HF organisation (text only) or the first auth…1960336 -
fix(save): saving from a trending card failed with a 500
siddhm11 +1: Claude Opus 5.5Trending cards matched by area, the week's top ("Most upvoted this week", "Trending this week") and every topic-page card have no interest, so their cluster_id is None. hx-vals rendered it as JSON null, htmx sends a null as the text "null", and the save handler's int("null") raised, so the Save button did nothing. Pass worked because app.js leaves nulls out of its forms. - action_buttons.html drops None values from hx-vals and data-vals (new drop_none filter), so a missing field takes the handler's default. - save, not-interested and unsave read cluster_id with _cluster_id(): a value that is not a whole number records no interest instead of failing, which keeps saves working from trending pages already open in browsers. Checked in a real browser on a local server: Save on an area-row card and on a "Most upvoted" card both return 200 and show Saved. Tests post exactly what htmx posts. Full unit suite: 1,491 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>0a5c9f3 -
feat(trending): Trending becomes a main tab, every topic gets a page, and new readers see trending in their areas first
siddhm11 +1: Claude Opus 5.5The owner's decisions on the Trending Tab Plan (doc 06, 2026-10-25): - Bars: Feed · Trending · Search · Library on desktop and phone. Collections moves inside Library, which gets its own tabs (Saved · Collections · Recent). With TRENDING_PAGE off the bars keep Collections, so nothing links to a page that is off. - /trending says which lists it shows ("Lists of 19–23 Oct · next list Mon 26 Oct" at weekends), leads with chips for rising and cooling topics and the reader's areas, gives shares in plain numbers too ("1 in 37 → 1 in 18"), puts the topics before the area rows, and lists the 10 named topics that fail the trend test as "Steady" with their share now, never as moving. - /trending/topics/<id> for every named topic: its 26-week line and label, its papers on this week's lists (papers listed since the daily grouping go to their nearest topic), its 10 most upvoted, recent arXiv papers nearest its middle paper (new qdrant_svc.search_recent on the recent shard, labe…408392e -
feat(trending): the shelf is always there, matching the areas a reader picked as well as their saves
siddhm11 +1: Claude Opus 5.5On 25 Oct the trending shelf vanished for a reader with 2 saves (after the Turso switch reset their history): 7 of the week's 312 Daily Papers fitted those saves, the first shelf showed 6, the next had 1 and hid, and at the weekend no new list came to refill it. The same week held 38 robotics, 118 vision and 137 AI papers by category. Doc 06, 2026-10-25. - A paper also qualifies by being listed in an area the reader picked (onboarding, /interests), scored below any save-close paper so saves lead and areas fill. Places go round the interests and areas, best first. - Papers shown before come after every unseen one instead of leaving the shelf; the feed page below it and saved/dismissed papers stay excluded. - A reader nothing fits gets the week's most upvoted, "Trending this week". - The feed requests the shelf on every tier, not only the clustered one. - refresh() stores each paper's arXiv categories (sidecar, then Turso). - /trending: a row per chosen area with its count of the …519bf40
2 days ago
-
fix(turso): stop three reads that scanned all 1.97M papers, and report a block
siddhm11 +1: Claude Opus 5.5Turso's free plan allows 500M rows read a month; on 2026-10-24 it blocked every read, even SELECT 1. * The figure warm-up found "papers from the last 2 days" by update_date, which has no index: every 6 hours and at every restart it read all 1.97M rows, about a quarter of the monthly allowance on its own. It now reads only the newest 5,000 rows by rowid, where the daily job appends new papers. * Without the sidecar, the starter feed's Turso fallback is a LIKE scan of every row. Local servers run without the sidecar unless told, so each new reader on a local server scanned Turso. The scan now runs only on the Space (SPACE_ID), where it means the sidecar failed to download, or with TURSO_SCAN_FALLBACK=on; otherwise it is skipped with a warning. * /healthz/deep reported Turso "ok" throughout the block: Turso answers HTTP 200 with the error inside the result. It now reads the result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>74dec34 -
fix(trending): /trending says so when topics could not be worked out
siddhm11 +1: Claude Opus 5.5With Turso unreachable the page claimed topics were "being worked out" for as long as the outage lasted. It now says they can't be worked out right now; the week's papers, which come from Hugging Face directly, still show. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>a572e3d -
feat(trending): rising topics from 26 weeks of Daily Papers, and a /trending page
siddhm11 +1: Claude Opus 5.5Once a day, trending_topics clusters the last 26 weeks of lists the daily job stores in Turso (hf_daily_papers, PR #56) the way §3.3 clusters a reader's saves: Ward on L2-normalised vectors from Qdrant, K = 24, medoids, and topic ids kept across days by stabilize_cluster_ids. A topic is rising or cooling only when the slope of its weekly share beats the same slope with every paper's week shuffled 2,000 times, Benjamini-Hochberg q < 0.05, and it is shown only under a phrase from its papers' titles, keywords and summaries that 30% of them carry. Phrases never cross a field; no model writes names (doc 06 defers LLM cluster summaries to Phase 8). On the 24 Oct data (5,176 papers): 30 s a day on a laptop, ids stable across recomputes, rising today: Skills, Robot, World Models. /trending lists the week's papers in the reader's interests, the rising and cooling topics with their weekly-share lines and top papers, and the week's most upvoted, with a footnote on the method; its cards log expo…e51c693 -
feat(search): trending papers step up in search for a few days (SEARCH_TRENDING)
siddhm11 +1: Claude Opus 5.5A paper in the top quarter of this week's Daily Papers that retrieval already found in its first 50 may take one of the last 2 places the cross-encoder scores. If it scores at least half the best, it gains up to 0.25 x the best score, in full for its first 3 days on the list and fading to nothing at 7; at most 2 papers per search. A moved result is tagged "Trending on Hugging Face" with its strongest fact; the classic card is byte-for-byte unchanged otherwise. Switch SEARCH_TRENDING: off (default), shadow (search_meta.trending_shadow records each would-be move, from and to, and nothing changes on screen) or on. No preview: search results are cached for everyone. Doc 06, 2026-10-24. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>d3ea811 -
feat(trending): each trending card says why, in counts from the lists
siddhm11 +1: Claude Opus 5.5Every shelf card now carries its place on the day it was listed ("#1 of 92 papers on 30 Sep") and, in the top quarter of a week of 20 or more, its place in the week ("top 1% of this week's 404"); relative places because the lists ran from 101 to 404 papers a week and upvotes are heavy-tailed (median 13, p99 271 over 26 weeks). Hugging Face's one-line summary is shown labelled as theirs, stars are shortened (5.7k), and comment counts are never shown: nearly every paper has one or two. The card moves to partials/trending_card.html so /trending can reuse it; its topic line and the shelf's /trending link render only once a later change attaches a topic. rank() and shelf_for() take a size, week_top() lists the week's most upvoted, and trending_now() hands search the week's papers with their freshness (full for 3 days, gone at 7). Doc 06, 2026-10-24. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>3c99209 -
fix(trending): read every page of a Daily Papers list, and what HF says about each paper
siddhm11 +1: Claude Opus 5.5The Daily Papers API serves at most 100 papers a page and stops at 50 without a limit, which cut 8 papers from 22 Oct 2026's 58. fetch_snapshot now asks for pages of 100 until a short one, and also keeps each record's keywords, one-line summary and comment count (`extra`) for the shelf's cards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>98d4c93
3 days ago
-
feat(search): a rewritten question searches keywords in the rewrite's words
siddhm11 +1: Claude Opus 5.5When the rewriter changes a query and no paper contains every word of the original (keyword search fell back to matching any word), the original's keyword list is left out of RRF. Its dense list and the exact-title boost stay. Keyword results now carry match="all"|"any" so the search can tell a question from a title. Switch SEARCH_REWRITE_KEYWORDS, off by default. Measured on the 79 distinct logged searches (top 10, blind 0/1/2 grades, rewrites pinned): 12 change, 124 -> 138 points, 6 better, 1 worse. All 9 known titles stay first. Without the guard the DPO paper fell out of the top 10, because the rewriter shortened its title. Rejected after testing, in doc 06: requiring the rarest words (LitSearch keyword top-10 33.9% -> 11.1%), phrases for hyphenated terms (neutral), and topping up short every-word lists with any-word matches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>b7cdd94 -
feat(feed): trending in your interests, a shelf of Hugging Face Daily Papers
siddhm11 +1: Claude Opus 5.5Readers in the clustered tier get up to 6 papers from Hugging Face Daily Papers (last 7 days, refreshed every 3 hours) in one block after the fourth card of page 1, loaded once the page is on screen. A paper qualifies when it is as close to one of the reader's interests as the 25th percentile of their own saves (clamped 0.55-0.70; 0.60 below four measurable saves); the order is closeness + 0.07 * log10(upvotes), each interest gets max(2, ceil(6/K)) places first, and fewer than 3 qualifying papers means no shelf. The shelf holds no feed slot, so quota, MMR and the served order are untouched (doc 06, 2026-10-23). Its cards carry their own query_id, propensity 1.0 and policy trending_shelf_v1 through links, Save and the exposure log; what it shows counts as shown for the feed and the next shelf. Interest names come from the feed page it sits on. Vectors come from Qdrant through the shared cache, so nothing is embedded on the Space. Switch TRENDING_SHELF (off by default); /healthz/runtim…727c02a -
fix(search): let long questions reach the query rewriter
siddhm11 +1: Claude Opus 5.5_looks_academic skips the rewrite when a query has more than six words and two "academic" matches. The acronym branch [A-Z]{2,} was compiled with IGNORECASE, so every ordinary word of two or more letters counted, and no query over six words has been rewritten since Phase 3 (d5a6f3e). "how do I stop a RAG system from making things up" counted 9 acronyms. The acronym branch is now case-sensitive; the topic words stay case-insensitive. Measured locally before and after on the 15 real search_log queries the change affects (top 10, graded 0/1/2 blind to which version found each paper): 171 -> 197 of 300; 4 better, 2 slightly worse, 9 unchanged. The RAG question goes from 2 to 15. Seven long paper titles still find their paper at #1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>d201982 -
feat(onboarding): describe your work in a sentence and get its areas picked
siddhm11 +1: Claude Opus 5.5A newcomer often can't map their work onto arXiv's area names. Step 1 now opens with "In a sentence": the description is searched (hybrid search with no LLM rewrite, so it reaches no model, and it is not stored), and the areas its top papers sit in are picked for the reader to check, with the evidence: "Picked Robotics, Computer Vision from 10 matching papers, like 'DexVIP: ...'". Picked areas carry a "from your words" tag, suggestions only add to what is already picked, and step 2 opens on papers for the sentence instead of popular ones. Doc 06 cites language-based preferences as competitive near cold start (Sanner et al. 2023). A paper's primary category counts 1 and a cross-list 1/2: cs.LG is cross-listed on so much that, counted fully, Machine Learning would be suggested for most descriptions. Probed on 12 descriptions with word-match search only (no BGE-M3 locally), 11 got sensible areas; "I like space stuff" matched COCO-Stuff. Logged in doc 06; CURRENT-STATE and CLAUDE.md updat…4c1781d -
feat(onboarding): a drawing of each research area on its card
siddhm11 +1: Claude Opus 5.5An area's name alone often means nothing outside the field: "Machine Learning" or "Condensed Matter" tells a newcomer little about what they would be choosing. Each of the 20 cards on /onboarding and /interests now shows the field doing its typical thing: a detector's box round a cat for Computer Vision, a line fitted to points with y = wx + b for Machine Learning, a question answered step by step for AI, a Feynman diagram, a Bell-pair circuit, the chord rule on an elliptic curve. The drawings are inline SVG written by scripts/build_area_art.py; the plotted ones (the curve, the helix, flow round a cylinder, the spectrogram) are computed. They take the page's colours in both themes and turn green with a picked card, and hover redraws the one annotated stroke (off under reduced motion). They are decorative to screen readers: the name and description still carry the meaning. Not images, because Hugging Face refuses binary files in pushed history. About 10 KB gzipped for all twenty. Co-A…ec199d7 -
fix(search): the query shows in the box, not as the heading
siddhm11 +1: Claude Opus 5.5The search page printed the query as its heading, in the box and in the result count ("10 results for ..."), and the empty state quoted it again. b804177 kept the heading in step with a search from the box: that fixed a stale heading but kept the repetition. The heading is now "Find papers", naming the page like every other masthead, and the count and the empty state no longer quote the query. The tab title still names it, so tabs and the browser's history tell searches apart. With the box the only place on the page that names the query, it has to be right after Back and Forward. htmx restores those from saved HTML, which holds the value the page was served with, not what was typed: Forward to a search run from the box showed its results under the first query. app.js now sets the box from the URL when htmx restores a page; the browser test fails on that line without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>d8d4da6 -
perf(feed): fetch candidate vectors and details side by side
siddhm11 +1: Claude Opus 5.5The multi-interest build fetched its candidates' vectors (Qdrant, in Europe) and then their details (the sidecar, else Turso, in Mumbai), one after the other, though neither needs the other. They now run together and the build waits for the slower one. The feed is unchanged: a test checks the same ranked list and tags whichever fetch finishes first. Measured 2026-10-23 from a laptop in India against the live services, 12 saves in 3 interests, four fresh processes each: vectors about 0.85 s, details 0.2-0.47 s, build 1.9-2.2 s either way; side by side the vector fetch slowed under contention, so the saving there was small and noisy. The Space is the case that matters: its own probe takes 978 ms for SELECT 1 to Turso (112 ms from the laptop), so there the details are likely the long half. feed_tier1 now reports both halves (cand_vectors_ms, cand_details_ms) next to their wall time (candidate_meta_ms), so the saving can be read off production. Co-Authored-By: Claude Opus 5.5 <noreply@an…6edee5d -
perf(feed): time every stage of the multi-interest build and the served page
siddhm11 +1: Claude Opus 5.5On the Space a first feed page for a reader with saves took 6.9 s (build_ms), and the stage timings then recorded covered 3.9 s of it: ann_retrieval 2.0 s, candidate_meta 1.8 s, clustering, MMR. The rest had no key, and candidate_meta folded two network fetches (candidate vectors from Qdrant, then their details from the sidecar or Turso) into one number. feed_tier1 now also records saved_vectors_ms, cluster_search_ms and st_search_ms (the two halves of ann_retrieval), cand_vectors_ms and cand_details_ms (the two halves of candidate_meta), features_ms, rerank_ms, exploration_ms and total_ms. A served page records feed_first_page or feed_next_page: page_ms, log_ms (impressions and exposures), cards_ms and request_ms, what the reader waited for. Timing only: the feed is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>f23a740
4 days ago
-
fix(search): the query rewriter never answers the reader
siddhm11 +1: Claude Opus 5.5On 2026-10-22 "HI HOW ARE YOU >" was rewritten as "I'm doing well, thank you! How can I help you today?", which was then searched beside the query and shown as "searched as". The prompt gave no rule for text that isn't a research topic, so the chat model answered it; the old prompt reproduces the reply word for word, and turns "hey whats up" into "general conversation". The prompt now says the text is a query, never a message, and a greeting comes back as is (Qwen: all four greetings tried came back unchanged, rewrites of real queries as before). A rewrite that talks back (you/your/thanks/I'm not in the query, a ? or ! the query didn't have, or over 12 words) is discarded; of the 39 rewrites logged in production only that one is caught. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>768a4ff -
fix(search): a search from the box renames the heading and the tab
siddhm11 +1: Claude Opus 5.5The search box swaps in only #search-results, so the page heading and the tab title kept the query the page was opened with: on 2026-10-23 the page read "how do I stop a RAG system from making things up" above the results for "HI HOW ARE YOU >". The htmx answer now carries the heading as an out-of-band swap and a <title> that htmx applies as the swap settles. The whole page is unchanged and has one heading. A browser test runs a second search from the box and checks the heading and title, then Back and Forward. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>b804177
5 days ago
-
fix(briefing): drop an over-long point instead of failing the whole reply
siddhm11 +1: Claude Opus 5.5In the first 25 papers of the b2 evaluation (2026-10-22), 24 briefings passed and one failed outright: 2609.31119's first point ran to 404 characters against the 400 limit, and the reply was thrown away with its two good points. A point or step over its limit is now dropped like one that fails the number check, and the over-long use line is recorded as dropped. Missing or non-text fields still fail the reply. The limits are unchanged: the other 66 points had a median of 220 characters and a longest of 326. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>721aff2 -
fix(briefing): send every section's opening and decline only when the paper says nothing
siddhm11 +1: Claude Opus 5.5In production the model declined 5 of the 15 papers it wrote for with {"insufficient": true}, all with 16-85 paragraphs; 4 of 5 reproduced. Passages were chosen by section title, so papers that name sections by topic ("Backbone", "Iterative Attention", "Cognitive Prompting") were sent only their introduction, and 3 paragraphs per top-level section sent 2608.11967 its experimental setup but not "Main Result Analysis". Now the picking is by place in the paper: abstract, introduction and conclusion, the first paragraph of every section and subsection, captions, then later paragraphs in rounds, the appendix last; never related work, references or front matter; sent in page order. The prompt asked for "the paper's numbers" behind a reply that opened on "insufficient", which let the model decline whenever no results were sent. Prompt b2 asks for numbers when the passages state them, counts a theory or survey paper's claims as its findings, and declines only when the passages don't say what…3223eb2 -
Merge origin/main into docs/session-2026-10-21
siddhm11Brings the harvest (#24, #41, #42) and the libsql fix (#45) under the redesign stack so the session's docs can be written against both.5938f71 -
feat(search): keep recent results in memory so going back doesn't search again
siddhm11 +1: Claude Opus 5.5Going back to a search ran the whole pipeline again: a BGE-M3 encode, a Groq rewrite, Qdrant and FTS, a metadata fetch, 1-5 s on the Space, and a second search in the analytics. app/search_cache.py keeps a query's ranked results in memory for SEARCH_CACHE_TTL_S (600 s; 120 s when a retrieval arm was missing, since a later run may do better), keyed by the query with case and spacing ignored, at most 256 queries. - Back, the paper page's back link, a reload and another reader's identical search are served from memory. - A reader coming back to the same results keeps the query_id they were shown (clicks stay tied to that search, section 3.11) and is not logged to search_log or counted in reader_days again. The reader-to-query link is memory-only and dies with the entry. - Callers get copies, so one reader's saved state or query_id never reaches another's results. Empty results are not kept. SEARCH_CACHE_TTL_S=0 turns it off. - /healthz/runtime counts search_cache_hit and sear…36c792e
6 days ago
-
fix(paper): the back link returns to where the reader came from
siddhm11 +1: Claude Opus 5.5The paper page showed "← Your feed" (to /) whenever the link carried a query id, and search results carry one too, so a paper opened from a search sent the reader to the feed. The link now follows the same-origin Referer to a known page: a search keeps its query (only `q`), and the feed, saved papers, recently opened, a collection or the previous paper get their own label. Without a usable Referer the click's surface decides; a shared link still gets no link. The browser test covers both halves: search, open a result, press Back, then use the page's own link. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>ba6d91e -
fix(search): Back to a search shows the whole page, not a bare fragment
siddhm11 +1: Claude Opus 5.5/search?q=... answers htmx with the results alone and a browser with the whole page, and nothing told the browser the two differ. After search -> open a paper -> Back, the browser could show the cached results fragment as the page: no header, no styles, no title (reproduced in Chromium against the sealed test world). - Every non-static response now carries Vary: HX-Request, and htmx fragments are sent with Cache-Control: no-store. - htmx restoring history without a snapshot (HX-History-Restore-Request) gets the whole search page; it swaps the answer in as the body. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>eeef7b6 -
fix(ingest): accept libsql:// Turso URLs
siddhm11 +1: Claude Opus 5.5Turso hands out libsql:// URLs, which urllib can't open. daily_refresh.py and refresh_citations.py already convert them to https://, but ingest_arxiv.py and ingest_backends.py did not, so the daily job worked only because the TURSO_URL secret was entered as https://. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>3e90cc0 -
fix(ask): fetch the paper's text before answering; don't cache abstract-only
siddhm11 +1: Claude Opus 5.5Ask read the same local-only text as the briefing. After a deploy it answered from the abstract alone, and a suggested question's answer was then cached for every reader. It now fetches the text first when this container lacks it, does not cache an answer written without it when the paper has paragraphs, and skips such a cached answer if one exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>ce7c67e -
fix(briefing): no "too few passages" for missing text; never downgrade
siddhm11 +1: Claude Opus 5.5After a deploy a briefing for a paper read earlier found only the abstract and stored "insufficient: too few passages" for good. Seen in production: 2508.16989 (61 stored paragraphs) and 2410.17725 (43). - The briefing fetches the paper's text again when it is missing here. If that fails, nothing is stored and a later visit retries (counted as no_text on /admin/ai). - A stored "too few passages" row is ignored when the source records enough paragraphs, so the two broken briefings repair themselves. - The cache key is paper + arXiv version + prompt. The parser version and model came out: citations link to arXiv's own anchors on a fixed version, so neither changes what a stored briefing says, and the parser bump to 2 had thrown good briefings away (2608.11967 went from ok to insufficient). Rows stored under the old key are still found. - A worse attempt never replaces a good briefing of the same version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>5771df0 -
fix(sources): fetch a saved paper again when its local text is gone
siddhm11 +1: Claude Opus 5.5paper_sources is replicated, but paper_fulltext and paper_paragraphs are local: a deploy (or prune past FULLTEXT_MAX_PAPERS) drops the text while the restored row still says "ok, current", so ensure() never fetched it again. has_text() says whether this container holds the text, and ensure(need_text=True) fetches when it doesn't. Opening a paper now asks for the text, so it is back before the briefing or Ask needs it; figure reads and the warm-up keep needing only the row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>2d52647 -
Merge pull request #44 from siddhm11/feat/paper-briefing
siddhm11Unverified signaturefeat(briefing): three things to know, how it works, where you'd use itadd0458 -
Merge pull request #43 from siddhm11/feat/paper-page-v2
siddhm11Unverified signaturefeat(paper): the redesigned paper page behind PAPER_PAGE_V25e6972b -
fix(daily): let catch-up runs raise the ingest limit; job ceiling 5 h
siddhm11 +1: Claude Opus 5.5The first catch-up run (2026-10-20) hit the 60-minute ingest limit after 2,414 papers: a GitHub runner encodes about 2,400 papers an hour, and about 15,000 were missing. A new `ingest_timeout_min` input (default 60, so scheduled runs are unchanged) is passed through env, and the job ceiling rises from 120 to 300 minutes. The ingest limit and per-call network deadlines remain the stall guards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>2007bc5 -
feat(daily): a `today` input for catching up in short windows
siddhm11 +1: Claude Opus 5.5The newest stored paper is from 2026-09-29. A single run with a 22-day lookback would encode every missed day at once on a CPU runner and overrun the ingest's 60-minute limit. Manual runs can now set `today` to a past day (passed as --today) and step it forward. The value reaches the script through env, never pasted into the shell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>0e6b4bd -
feat(daily): harvest through OAI-PMH and report licence coverage
siddhm11 +1: Claude Opus 5.5stage_ingest passes --source oai, so the daily job uses arXiv's OAI-PMH feed instead of per-category search queries. The summary gains a licences block: new papers with a licence row, how many are open, and how many may show figures under FIGURE_LICENCES. Reported only; it never fails the run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>c36035c -
feat(ingest): keep each harvested paper's licence in Turso paper_licences
siddhm11 +1: Claude Opus 5.5parse_oai_record reads <license> into licence_url and a short code (app.licences.code_from_url). New papers' licence rows go in the same Turso pipeline as their papers rows; papers already stored get theirs when OAI lists them again. Own table, never a papers column: INSERT OR REPLACE rewrites whole papers rows (trap T5). The upsert skips rows that already say the same, so daily re-harvests write nothing. Dry run over 2026-10-16: 1,833 papers kept, all with a licence (ARXIV-NONEXCLUSIVE 865, CC-BY-4.0 773, CC-BY-NC-ND 79, CC-BY-NC-SA 67, CC-BY-SA 36, CC0 13). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>da423eb -
Merge remote-tracking branch 'origin/main' into feat/daily-harvest-licences
siddhm1138e1da6 -
feat(ingest): harvest new papers through arXiv OAI-PMH
siddhm11 +1: Claude Opus 5.5arXiv's OAI-PMH interface is built for incremental harvesting: everything changed between two dates, a whole archive per request stream, about 1,300 records a page. A day across our seven archives is about 10-15 requests, where the search API needs one query per category (about 100), which is what got the daily run throttled on 2026-10-01. --source oai harvests each archive set holding our categories, keeps new- style ids from the month before --since (OAI's <created> is not the first version's date for revised papers), and reports a set complete only when its last page carries no resumption token (exit 3 otherwise). Text is NFKC- normalised and arXiv's alias categories collapsed as the search API does: 901 overlapping papers from 2026-09-29 matched rows written from the search API exactly. Requests identify themselves; a 406, like a 403, 429 or 503, is a temporary block to wait out, and other 4xx answers fail at once. Written 2026-10-01 and left uncommitted; committed unchanged. Co…3f01cfe -
feat(admin): readers, searches and AI history dashboards
siddhm11 +1: Claude Opus 5.5Three owner-only pages next to AI usage, with tabs across all four: /admin/readers readers today, this week and month, the share who came back, what they did, new and returning readers per day, and readers per month since launch with where saves came from /admin/searches searches per day with and without results, the share that led to an opened or saved paper, the most searched queries, queries that found nothing, and the latest searches with their AI rewrite and overview /admin/ai/history every AI call by paper (title, plain summary, briefing outcome, tokens) and by search (rewrite, overview) The owner's own use is left out unless 'Include my own use' is chosen. Model-written text is shown as text, never markup. Every chart now comes from one builder with one tooltip script: whole-number ticks for small counts, about seven l…e8530b8 -
feat(analytics): count what each reader does, per day
siddhm11 +1: Claude Opus 5.5There are no accounts, so a reader is the anonymous cookie. Interactions recorded saves and passes, but not visits, searches, feed pages, papers opened or questions asked, and the owner's own testing was mixed in with everyone else's (October's three active readers were the owner). reader_days keeps one replicated row per reader per UTC day: pages, searches, feed pages, papers opened, saves, passes and questions, with owner = 1 for a browser holding the preview token. analytics.observe(), called from the HTTP middleware for every finished request, classifies it by route, so no handler has to remember; failed requests, first visits, static files, figures, health and admin pages don't count. Counts live in memory and are written every minute and on shutdown, rows rewritten whole so turso_sync pushes them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>ae5214c -
feat(analytics): log searches, and keep and reuse their AI overviews
siddhm11 +1: Claude Opus 5.5Searches weren't recorded at all: 321 of 335 saves came from search results, but not how many searches there were or what was asked. search_log now keeps one row per search: the query (collapsed, at most 200 characters), its query_id (which the results' clicks and saves carry), the result count, retrieval mode, the AI rewrite when it changed the query, the time taken, and whether it was the owner. It is not stored against the reader. search_overviews keeps the AI overview shown for a query over its top five papers, and is also its cache: the same search reuses it for a week instead of a new model call (counted as search_overview cache hits). What people typed is kept 90 days: a daily housekeeping run clears query text, rewrites, overviews and search subjects in llm_calls, locally and in Turso, which only ever receives new rows and so must be told. Counts stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>bc1e7b6 -
feat(ai-usage): record the paper or query each AI call was for
siddhm11 +1: Claude Opus 5.5llm_calls gains a replicated subject column: the arXiv id for plain summaries, briefings and Ask, and the search query (whitespace collapsed, at most 200 characters) for the search rewrite and overview. A call that failed or found every model busy now says which paper or search it was for. Search queries are cleared after 90 days by the next commit's housekeeping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>600f214 -
feat(admin): an AI usage dashboard at /admin/ai
siddhm11 +1: Claude Opus 5.5/healthz/ai is data for machines; this is the same data for a person, visible only with the preview token (404 for everyone else, noindex): today model calls, tokens, share answered by a backup model, share served from the cache, failed or busy calls stands out plain sentences with a status icon and word: the feature using most tokens (and whether it is cached), a model near its daily tokens or its minute, backup answers over a tenth, busy or failed calls, cache hit rates, declined briefings, reports models each model's tokens today against 200,000, and the requests and minute-tokens Groq last reported, as meters labelled when close features per feature: calls, answered, backup, cut off, failed, tokens and time per call, cache hit rate, every outcome count history tokens per day for 14 days, stacked by feature, one tooltip per day listing every feature, and th…61a86da -
fix(ai-usage): give a rate-limit window's age, not its raw timestamp
siddhm11 +1: Claude Opus 5.5/healthz/ai listed both the raw 'at' timestamp and age_s for each model's window: the timestamp was removed from the original after the copy was made. Seen on the live page after the first real call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>279a374
7 days ago
-
feat(health): /healthz/ai shows where the AI goes today
siddhm11 +1: Claude Opus 5.5Per feature: calls, ok, fallback (a model other than the primary answered), truncated, failed, unavailable, unparsable and over-budget calls, tokens in and out, average time, and the day's counts (cache hits, outcomes, drops, reports), including counts not yet written. Per model: calls, tokens and their share of the day's 200,000-token limit, and the newest rate-limit window Groq reported. Plus the briefing budget and 14 days of history. Groq's limits per model are now config (GROQ_REQUESTS_PER_DAY, GROQ_TOKENS_PER_DAY, GROQ_TOKENS_PER_MINUTE): no header reports tokens per day, so that limit has to be known to be compared against. Counts only: never a prompt, an answer or a reader. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>359b725 -
feat(ai-usage): count cache use and answer quality per feature, per day
siddhm11 +1: Claude Opus 5.5Calls alone don't say how well the AI did. ai_daily keeps one number per UTC day, feature and metric, replicated: explain cache_hit, cache_miss, written, trimmed, declined search_rewrite skipped (already academic), rewritten, unchanged, discarded search_overview shown, trimmed, empty briefing cache_hit, cache_miss, ok, insufficient, failed, busy, points_dropped (by the number and anchor checks) ask cache_hit, cache_miss (suggested questions), answer, unanswerable, nosupport, cap, busy, error, quotes_kept, quotes_dropped, cites_dropped any feature reported (Report a problem) ai_usage.count() only adds to a dict; a loop owned by the app lifespan writes the counts every minute and on shutdown, each row rewritten whole so turso_sync pushes it, and keeps them if a write fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>94b838e -
feat(ai-usage): log search rewrites, search overviews and plain summaries
siddhm11 +1: Claude Opus 5.5These three call Groq directly and logged nothing: no tokens, no failures, no sign of which model answered. groq_svc._run_logged now runs each one off the event loop and writes its llm_calls row, as search_rewrite, search_overview or explain, including calls that failed or found every model benched (status unavailable). Each feature still degrades exactly as before; a search that already looks academic makes no call and logs none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>134608d