Spaces:
Running
Running
Commit History
2 days ago
-
docs(06): the Lasting page's views, filters and the hidden-gem rule
siddhm11 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>b72a1df -
docs(06): Hacker News as a second signal, with the measurements behind it
siddhm11 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>855b8b9
6 days ago
-
feat(search): a short search that names a paper shows that paper first
siddhm11 +1: Claude Opus 5.5"resnet", "lora", "gpt-3 paper" name one paper whose title lacks the name, and later papers that use it fill every retrieval list: 2 of 30 such originals reached page 1. For searches of up to 4 words the primary model is asked for the paper's exact title, in a call of its own beside the rewrite; if a corpus title matches almost exactly (difflib >= 0.92, most cited on a tie) that paper goes first, labelled "Best match for the name you searched". Measured 2026-10-30 on rule-built sets (doc 06): 302 named searches, #1 60 -> 132; 140 held-out names, 47 -> 64, with 6 pins of another paper (mostly a sibling, 2 displaced a correct #1); 7 of 238 topic searches got the field's canonical paper pinned. That narrowly misses the pre-registered limits (4.3% wrong on the held-out names, 5.0% topics); see doc 06. - app/named_paper.py: gate, strict title check, pin; waits at most 1.6 s from the start of the search (Qwen p50 124 ms), never blocks it. - groq_svc.name_paper asks only the first configu…fddee41 -
perf(search): run the search models without holding the GIL
siddhm11 +1: Claude Opus 5.5Every torch op called from Python retakes the GIL. While any other thread runs pure Python, each retake waits up to the switch interval (5 ms), so a search stalled for seconds with the CPU idle (Space, 26 Oct: "bert paper" 9.3 s). On the app's own search path with an arXiv HTML parse looping in the background: 11.0 s median today, 0.37 s with this change, 0.35-0.39 s with nothing in the background. Top-10 lists identical (20/20). - app/model_trace.py traces BGE-M3's transformer and the cross-encoder with torch.jit at load (DeBERTa's masked softmax as plain ops, so no Python op is left in the graph), checks each trace against the eager model on probe inputs and falls back to eager if it fails or differs. Outputs were bit-identical on 400 search pairs and 40 encodes. Switch SEARCH_MODEL_TRACE: off (default), shadow, on. - compute_pool.configure_gil() applies GIL_SWITCH_INTERVAL_MS (0 keeps Python's default; 0.5 recommended) for the GIL retakes that remain. - /healthz/runtime…4e07a2d
11 days ago
-
fix(groq): keep cached explanations, let a key take concurrent requests, check routed rewrites like direct ones
siddhm11 +1: Claude Opus 5.5Review of the key-rotation commit before release found four defects: - The explanation cache key gained the title. With failover off, every cached explanation would miss once after a deploy (a Groq call each), and feed cards, which read the old key in card_data, would never show one written after it. The title is out again: for the head of the chain the key is the old key, and cards also read a fallback model's entry. - A key served one request at a time. With the one key there is, every Groq call queued: a 2 s rewrite waited behind an 8 s explanation and came back unrewritten. Groq limits requests and tokens per minute, not connections; only a recovering scope is now held to one probe. - The routed rewrite skipped _reads_like_a_reply, so a rewrite that answers the reader would have been searched. - It also refused rewrites over 8 words that the direct path accepts, spending a GPT-OSS call on them. Live check with the local key (scripts/check_groq_routes.py --probe …60e843f -
feat(groq): rotate credentials before falling back to another model
siddhm11(cherry picked from commit a40fa57bed16db9744167f921373506f1efb3fb6)a84b508 -
docs(trending): a deploy keeps what was read; only thumbnail files go
siddhm11 +1: Claude Opus 5.5paper_sources is replicated to Turso, so the record of read papers survives a deploy (191 after the 25 Oct deploy, up from 187). The earlier note said every deploy re-read the week; only the WebP files on disk are lost, and /fig remakes each on its first view. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>8819ef6 -
docs(06): trending cards show the paper's first figure; papers before topics
siddhm11 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>674fd93 -
feat(trending): Trending becomes a main tab, every topic gets a page, and new readers see trending in their areas first
siddhm11 +1: Claude Opus 5.5The owner's decisions on the Trending Tab Plan (doc 06, 2026-10-25): - Bars: Feed · Trending · Search · Library on desktop and phone. Collections moves inside Library, which gets its own tabs (Saved · Collections · Recent). With TRENDING_PAGE off the bars keep Collections, so nothing links to a page that is off. - /trending says which lists it shows ("Lists of 19–23 Oct · next list Mon 26 Oct" at weekends), leads with chips for rising and cooling topics and the reader's areas, gives shares in plain numbers too ("1 in 37 → 1 in 18"), puts the topics before the area rows, and lists the 10 named topics that fail the trend test as "Steady" with their share now, never as moving. - /trending/topics/<id> for every named topic: its 26-week line and label, its papers on this week's lists (papers listed since the daily grouping go to their nearest topic), its 10 most upvoted, recent arXiv papers nearest its middle paper (new qdrant_svc.search_recent on the recent shard, labe…408392e -
feat(trending): the shelf is always there, matching the areas a reader picked as well as their saves
siddhm11 +1: Claude Opus 5.5On 25 Oct the trending shelf vanished for a reader with 2 saves (after the Turso switch reset their history): 7 of the week's 312 Daily Papers fitted those saves, the first shelf showed 6, the next had 1 and hid, and at the weekend no new list came to refill it. The same week held 38 robotics, 118 vision and 137 AI papers by category. Doc 06, 2026-10-25. - A paper also qualifies by being listed in an area the reader picked (onboarding, /interests), scored below any save-close paper so saves lead and areas fill. Places go round the interests and areas, best first. - Papers shown before come after every unseen one instead of leaving the shelf; the feed page below it and saved/dismissed papers stay excluded. - A reader nothing fits gets the week's most upvoted, "Trending this week". - The feed requests the shelf on every tier, not only the clustered one. - refresh() stores each paper's arXiv categories (sidecar, then Turso). - /trending: a row per chosen area with its count of the …519bf40
12 days ago
-
docs(06): trending v2 deployed; the search step-up goes straight to on
siddhm11 +1: Claude Opus 5.5A shadow replay of 99 searches (84 logged, 15 on this week's trending subjects) moved no result; four already-ranked trending papers would be tagged and two auditions stayed out below half the best score. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>82e7d11 -
docs(06): trending v2 — why a paper trends, rising topics, a few days' step-up in search
siddhm11 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>cba4898
13 days ago
-
feat(search): a rewritten question searches keywords in the rewrite's words
siddhm11 +1: Claude Opus 5.5When the rewriter changes a query and no paper contains every word of the original (keyword search fell back to matching any word), the original's keyword list is left out of RRF. Its dense list and the exact-title boost stay. Keyword results now carry match="all"|"any" so the search can tell a question from a title. Switch SEARCH_REWRITE_KEYWORDS, off by default. Measured on the 79 distinct logged searches (top 10, blind 0/1/2 grades, rewrites pinned): 12 change, 124 -> 138 points, 6 better, 1 worse. All 9 known titles stay first. Without the guard the DPO paper fell out of the top 10, because the rewriter shortened its title. Rejected after testing, in doc 06: requiring the rarest words (LitSearch keyword top-10 33.9% -> 11.1%), phrases for hyphenated terms (neutral), and topping up short every-word lists with any-word matches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>b7cdd94 -
feat(feed): trending in your interests, a shelf of Hugging Face Daily Papers
siddhm11 +1: Claude Opus 5.5Readers in the clustered tier get up to 6 papers from Hugging Face Daily Papers (last 7 days, refreshed every 3 hours) in one block after the fourth card of page 1, loaded once the page is on screen. A paper qualifies when it is as close to one of the reader's interests as the 25th percentile of their own saves (clamped 0.55-0.70; 0.60 below four measurable saves); the order is closeness + 0.07 * log10(upvotes), each interest gets max(2, ceil(6/K)) places first, and fewer than 3 qualifying papers means no shelf. The shelf holds no feed slot, so quota, MMR and the served order are untouched (doc 06, 2026-10-23). Its cards carry their own query_id, propensity 1.0 and policy trending_shelf_v1 through links, Save and the exposure log; what it shows counts as shown for the feed and the next shelf. Interest names come from the feed page it sits on. Vectors come from Qdrant through the shared cache, so nothing is embedded on the Space. Switch TRENDING_SHELF (off by default); /healthz/runtim…727c02a -
feat(onboarding): describe your work in a sentence and get its areas picked
siddhm11 +1: Claude Opus 5.5A newcomer often can't map their work onto arXiv's area names. Step 1 now opens with "In a sentence": the description is searched (hybrid search with no LLM rewrite, so it reaches no model, and it is not stored), and the areas its top papers sit in are picked for the reader to check, with the evidence: "Picked Robotics, Computer Vision from 10 matching papers, like 'DexVIP: ...'". Picked areas carry a "from your words" tag, suggestions only add to what is already picked, and step 2 opens on papers for the sentence instead of popular ones. Doc 06 cites language-based preferences as competitive near cold start (Sanner et al. 2023). A paper's primary category counts 1 and a cross-list 1/2: cs.LG is cross-listed on so much that, counted fully, Machine Learning would be suggested for most descriptions. Probed on 12 descriptions with word-match search only (no BGE-M3 locally), 11 got sensible areas; "I like space stuff" matched COCO-Stuff. Logged in doc 06; CURRENT-STATE and CLAUDE.md updat…4c1781d
15 days ago
-
docs(06): log the briefing passage and prompt change
siddhm11 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>a7c3c90 -
docs(06): log the harvest, briefing-text and back-to-search decisions
siddhm11 +1: Claude Opus 5.5Three changelog entries for 2026-10-20/21: - daily papers through arXiv OAI-PMH, licences in their own Turso table, with the measured runner throughput and the 23,098-paper catch-up; - a paper's local text is fetched again, and briefings are keyed on paper version and prompt (why not the passages sent); - Back to a search: Vary on htmx URLs, a true back link, and recent results in memory, including the query_id rule for a reader's repeat and why clearing search_log.query_id was dropped. CLAUDE.md: corpus numbers (arxiv_recent 279,778; Turso 1,876,875 to 2026-10-20), the search cache rule, two integrity rules (a replicated row doesn't mean local data is here; one URL, two bodies needs Vary), the query_id note in 3.11, the test count and the dates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>4986928
16 days ago
-
docs: record reader and search analytics and how queries are kept
siddhm11 +1: Claude Opus 5.5CURRENT-STATE gains a reader and search analytics row and lists the three new replicated tables. Doc 06 logs the decision: searches logged, query text kept 90 days and never stored against a reader, the owner's own use flagged and left out, with the production numbers that made it necessary (321 of 335 saves from search, no search ever recorded; 7 of 131 readers came back). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>c89cdb2
17 days ago
-
docs: record AI usage measurement and the caching flaws it found
siddhm11 +1: Claude Opus 5.5CURRENT-STATE gains an AI usage row (llm_calls columns, ai_daily, /healthz/ai, Groq's limits) and lists ai_daily as replicated; the LLM row no longer says the Groq chain is undeployed. Doc 06 logs the decision with the token arithmetic behind it and three findings from production: the briefing cache key thrown away by an unrelated parser bump (one good briefing replaced by a decline), uncached search AI, and plain summaries written twice. A schema comment still said 8 figures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>5af37f8 -
docs: record arXiv's access rules, the background lane and search thumbnails
siddhm11 +1: Claude Opus 5.5Doc 06 logs the split between the readers' pacer and a 15 s background lane, with the sources: arxiv.org/robots.txt (Crawl-delay: 15), arXiv's bulk-data page (interactive use first) and its API terms, which also leave figure redistribution to the copyright holder; that licence risk is recorded as the owner's open call. CURRENT-STATE gains the search thumbnail row and the warm-up's shown-papers queue, lane, cap and batch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>d0e4419 -
docs: record vector figures, the 24-figure cap and the figure rate limit
siddhm11 +1: Claude Opus 5.5CURRENT-STATE's figure row now covers SVG files and inline drawings, how they are cleaned and served, reduced-scale JPEG decoding, the 24-figure cap, the 600-a-minute limit and the hidden broken figure. Doc 06 logs serving cleaned SVG instead of rasterising (no native dependency, three layers keep script out) with the 50-paper measurement behind it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>7c5c922 -
docs(feed): record the two-column feed, layout logging and figure warm-up
siddhm11 +1: Claude Opus 5.5CURRENT-STATE gains rows for the v2 card, the wide feed, the layout column and ui_events, and the figure warm-up worker. Doc 06 logs the decision that the two columns keep served order (grid, auto-flow row, never dense) and that interactions record their layout, so position analysis can be split by it. CLAUDE.md §3.1 and §3.11 carry both as guards. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>a0b3515
18 days ago
-
feat(ask): an evaluation script, and log Ask in doc 06 and CURRENT-STATE
siddhm11 +1: Claude Opus 5.5scripts/eval_ask.py asks each paper the three suggested questions and one it almost surely doesn't cover, plus any owner-written ones, and writes a page to rate the answers with the gate's numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>1a6bc6f -
docs: log the paper briefing in doc 06 and CURRENT-STATE
siddhm11 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>2300f36 -
docs: log the paper source layer and figure licence decision in doc 06
siddhm11 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>f67c050
20 days ago
-
docs: record the FTS rank-first fix and feed policy v11 in doc 06
siddhm11 +1: Claude Opus 5.5Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>ecdedfa
21 days ago
-
docs: record the search reranker swap and its correction in doc 06
siddhm11 +1: Claude Opus 5.5Doc 06 changelog: 2026-10-14 (mxbai-rerank-xsmall, load-aware depth, filler-free FTS, split pools) and 2026-10-15 (gain re-measured on production's truncated abstracts: +7.7%, not +10-12%; idle depth back to 10; model pins; io pool cap). CLAUDE.md section 3.4 and the quick reference, PHASE8 stage table, README and CURRENT-STATE updated to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>7550b06
27 days ago
-
feat(reranker): rescale profile similarities, weight recency 0.45
siddhm11 +1: Claude Opus 5.5Raw profile similarities barely vary across a reader's candidates, so recency and position decided the feed's order. The personalised branch now rescales long- and short-term similarity across the candidate set (as cold start already did) and weights recency 0.45, option C of a measured relevance/freshness frontier chosen by the project owner. Cold start keeps 0.15. Against the old formula: held-out right answer in the top 10 55.1% -> 68.4% (272 simulated readers), real-reader AUC 0.684 -> 0.703 (+0.019, CI +0.011 to +0.027, 526 readers), recent papers in the top 10 43% -> 28%. The installed code reproduces the tested reference (identical metrics at pool sizes 30/60/100; one 1.5e-6 score tie in 770 top-10s). Updates CLAUDE.md section 3.4 weight budget; logged in doc 06 changelog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>fd260fc -
fix(reranker): age papers from first publication, not last revision
siddhm11 +1: Claude Opus 5.5Every age-derived feature (age, recency, year difference, is-newer, citations per year) now uses the month of first submission decoded from the arXiv ID. `published` in a paper dict is update_date, the latest revision, so revised classics were scored as new: Attention Is All You Need (2017, revised 2023) got recency 0.098 instead of 0.001. Validated before shipping on 272 citation-built readers: +3.3 pts top-10 hit rate (95% CI +1.5 to +5.5; 9 better, 0 worse) with no loss of recent papers in the top 10; neutral on 526 real readers' votes. The installed code reproduces the tested reference top 10 for every reader. Logged in doc 06 changelog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>3a065fb
-
docs: record the Llama 3.3 retirement and the Groq model chain
siddhm11 +1: Claude Opus 5.5- Doc 06 changelog: why the model is an ordered, monitored chain, why a Preview model can lead it, and the measured order. - CURRENT-STATE: a contract row for the LLM features, with the failure path and the health signal. - PHASE8: stage 1 timing now notes the model and the truncation rule. The 161-329 ms figure was measured on the retired Llama. The changelog also records a pre-existing bug found while testing. _looks_academic counts every word as an acronym under IGNORECASE, so a query of more than six words is never rewritten. It is noted, not fixed: changing it changes search behaviour and Groq usage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>7276ed9
-
docs: log the 2026-09-30 multi-interest and starter-feed decisions
siddhm11 +1: Claude Opus 5.5Doc 06 changelog entry for the four approved changes, with the replay measurements behind them; CLAUDE.md sections 3.3, 3.5 and the quick reference, and the CURRENT-STATE contract rows, updated to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>ce8a24c
-
docs(research): log the fresh-discovery and shadow-source decisions
siddhm11 +1: Claude Opus 5.5Doc 06 changelog entries for 2026-09-24 (fresh discovery, explicit interests, recoverable reading journeys) and 2026-09-25 (separating source candidates from relevance evidence; the first HF shadow milestone). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>9bb025c
-
docs(backfill): measured estimate, and the sidecar caveat that reframes it
siddhm11 +1: Claude Opus 5Repairing Turso does not reach production by itself. turso_svc.fetch_metadata_batch reads LRU -> local sidecar -> Turso, and the sidecar carries abstract_preview for all 1,799,348 rows, so Turso is essentially never consulted for metadata in production. It is read-only and baked into the image from a pinned HF dataset revision. So the backfill is step 1 of 4: rebuild the sidecar, upload it, repoint METADATA_SIDECAR_URL, redeploy. Worth writing down because the obvious mental model — fix the database, the app shows the fix — is wrong here, and acting on it means a long job with no visible result. The estimate is now measured rather than modelled. Three earlier figures were wrong: 13.6h counted only the arXiv pause; 22.6h added a guessed write cost but kept the inflated 90.6% truncation count; 10h fixed the count but kept the guess. Real batches give ~7.75s per 200 rows, so ~15.5 hours for the remaining 1,436,005. Validated before launching: 400/400 fetched, zero missing (the old-styl…4f06b54
-
fix(arxiv): old-style ids were collapsing onto their category
siddhm11 +1: Claude Opus 5_normalise_id matched [^\s/v]+, which stops at a slash. Old-style arXiv ids carry a category prefix — math/0309136v1, hep-ph/0512038v2, acc-phys/9502001 — and 115,604 papers in the corpus (6.4%) use that form. Every one normalised to just its category: math/0309136v1 and math/0511124v2 both became "math". Two silent consequences. A lookup by the real id missed, so the arXiv metadata fallback returned nothing for 6.4% of the corpus. And distinct papers collided on one key, so one paper's metadata could be returned for another — the same defect class this codebase already documents for Qdrant point ids. The character class also treated a literal 'v' as a terminator. Rewritten procedurally: strip a known prefix, strip a trailing version. Two independent operations, and one regex doing both is what hid this. None of the existing parametrised cases covered an old-style id, which is why it survived. Six now do, plus a collision test and a §3.9 string-type test. Found while writing the ba…da80710 -
feat(ops): make swallowed exceptions diagnosable, and size the abstract repair
siddhm11 +1: Claude Opus 5Two things needing attention, both measured rather than assumed. ERRORS grep -rn "traceback|exc_info" app/ returned ZERO hits across the whole application, against 95 `except Exception` handlers. Every production failure was a single line with no traceback and no line number. Worse, the handler that reports feed failures sits at the bottom of a 473-line try covering clustering, quota, retrieval, metadata, reranking, MMR and labelling — so "multi-interest preprocessing failed: KeyError('x')" narrowed the fault to "somewhere in the feed" while the user silently dropped to Tier 2. Swallowing is the right CHOICE there: a feed that degrades beats one that 500s, and those paths are deliberate. Swallowing SILENTLY was not. app/errors.py adds report(), which prints the exception type, the failing file and line, a traceback trimmed to OUR frames — an untrimmed trace through httpx and asyncio buries the one frame that matters — and any chained cause, which is where the answer usually is. Sti…43e5407 -
feat(paper): put comprehension on the page that was built for it
siddhm11 +1: Claude Opus 5Two surfaces, one LLM-free and one generated. The paper page existed and was empty; this is what it was for. READING DENSITY (app/readability.py) A three-band badge from notation rate, acronym rate, long-word share and sentence length. No model, no network, microseconds — cheap enough to compute inline anywhere. It deliberately does NOT claim to measure how hard a paper is to UNDERSTAND. doc 01 names difficulty ratings as an unfilled gap and also says why the obvious approach fails: Flesch-Kincaid and FORCAST are calibrated on general prose and mislead badly on academic writing, where a conceptually simple paper can score unreadable. What IS honestly computable from text is how dense the abstract is to read, so the wording is "Dense with notation", never "hard". Thresholds are the terciles of the real corpus (n=525), not round numbers. My first guess of 0.34/0.50 put 82.5% of papers in a single band — a label that never discriminates costs the reader attention and returns nothing. …c3db102 -
feat(events): log the click-through, and stop discarding rank 0
siddhm11 +1: Claude Opus 5app/db.py has declared `click` as a valid event_type since the schema was written and nothing ever wrote one. The system knew which papers were saved and which were dismissed, but not which were OPENED — the difference between "scrolled past" and "read", and the densest engagement signal a feed produces. Phase 7 cannot compute per-feed CTR without it. The card title now carries the provenance it was served with (qid/pos/src/prop/pol) and /p/{id} records it, so a click ties back to the exact feed request that produced it. A bare visit — a shared link, a bookmark, a crawler — logs nothing, because it genuinely has no propensity and no policy, and recording one as though it did would corrupt the §3.11 analysis rather than serve it. Found while testing that: `position or None` was discarding rank 0. Position 0 is the TOP of the feed, the most-clicked slot and the one any CTR-by-rank analysis depends on most, and it was being stored as NULL — indistinguishable from "no position at all". A…6cc8a37 -
feat(paper): give a paper a home on this site
siddhm11 +1: Claude Opus 5No URL on this site was ABOUT a paper. Every card sent its title straight to arxiv.org, so the product had nothing of its own to share, to link to, or to return to. Three structural consequences: - Nothing was shareable. The Open Graph tags added earlier could only describe the feed, because the feed was all there was to describe, so a pasted link never carried a paper's title. - There was nowhere to put comprehension. Plain-language explanation, difficulty, "read this first" — each needs a surface, and a card in an infinite feed is not one. - Every session ended at arxiv.org. GET /p/{arxiv_id} serves full metadata, the abstract unclamped (a reader who opened this page chose to stop), per-paper og:title / og:description / og:type=article, links to the PDF and the 3D map, and a "What sits near this" section. That section is drawn from the paper's OWN embedding, not the reader's profile. The distinction is the point: it answers "what is this paper next to", which…596fcec -
fix(ui): a phone had no way to change theme
siddhm11 +1: Claude Opus 5`.nav` is display:none below 720px and the theme toggle lived inside it, so the control was unreachable on mobile — on a product whose whole premise is reading on a phone. Moved into the top bar rather than the bottom nav: the bottom nav is a four-item grid and a fifth item would unbalance it, while the top bar has nothing else on the right at mobile widths. It carries its own margin-left:auto below the breakpoint, since that was inherited from .nav. Also records the batch of signal-integrity and resource fixes in doc 06 and adds CLAUDE.md §3.6b, so the invariants they establish are stated where the next change will look for them. Verified by the real-browser suite, which exercises both themes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>02009a6 -
fix(profiles): the long-term profile was one paper, and most updates were lost
siddhm11 +1: Claude Opus 5The long-term profile is the largest term in heuristic_score (0.40), the relevance axis MMR selects against, and the whole of Tier 2. The UI calls it "the overall profile built from your saved papers". It was neither built from them nor overall. Three independent causes, none of which is a change to alpha — the steady state remains the documented EWMA. 1. STARTUP. A plain EWMA seeded with its first observation never escapes it. The profile was L2-normalised after every step, so it always had unit magnitude and each new paper got a fixed 3% pull against a full-strength incumbent. Measured over near-orthogonal saves, cos(profile, first_save) was 0.996 after ten saves and still 0.982 after FORTY. Fixed with a running-mean warmup, alpha_eff = max(alpha, 1/(count+1)), which converges to exactly the documented alpha at 33 saves, continuously — the two rates are equal where they cross. 2. NORMALISATION. The magnitude of a running mean of unit vectors is the signal…c9f4cae -
fix(recommend): make the multi-interest feed actually multi-interest
siddhm11 +1: Claude Opus 5Three separate mechanisms were each independently collapsing the feed to a single interest. Each one was only visible once the previous was fixed, and none was detectable from a unit test of any single stage — every stage was individually correct. 1. merge_quota_results concatenated clusters instead of interleaving them. The quota was honoured in the pool, so no interest was starved, but every minority paper landed after the dominant block. compute_features sets candidate_position from the index in that list and feature 35 (position_inverse) carries a 0.10 weight in heuristic_score — the live scorer — so fusion order was laundered into a relevance signal. Now interleaved on a proportional stride schedule; strict round-robin would over-serve a 25% interest just as badly as blocks under-served it. 2. Quota was never enforced on the SERVED order. rerank_candidates sorts globally and MMR selects greedily; both are cluster-blind, so a correct 50/50 pool was serv…b648634
-
docs: record the cold-start churn design and the trending recency bug
siddhm11 +1: Claude Opus 5Adds §3.4b to the rulebook (impression memory, why propensity must never go back to 1.0 on a randomised tier, why epsilon-greedy over Plackett-Luce here) and two dated entries to doc 06's changelog, including the Phase 9 scope tension and the measured before/after. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>c95cd95
-
docs: record the RRF decision and the shard gap instead of re-deciding them
siddhm11 +1: Claude Opus 5Two open questions from the audit, logged per CLAUDE.md §6.2/§6.3 so they are not re-litigated from scratch next session. RRF: keep it, but stop calling it settled. OpenSearch's BEIR comparison puts rank-based RRF 3.86% below tuned score normalization on average and 4.81% below on SciDocs, the closest dataset to this corpus, and the "RRF is cheaper" defence does not survive the same table (1.62% at p50, -0.033% CPU). That is not enough to switch: normalization only wins *tuned*, and the only ground truth here is a 60-query known-item set, so tuning now would overfit to 60 queries. Section 3.1 keeps the rule and loses the claim that it is the higher-relevance choice. The sharper reading of that table is recorded too: search runs the dense arm alone today, and dense-only scores 0.1075 on SciDocs against 0.1602 for the best hybrid. The missing sparse arm is a much bigger lever than the fusion strategy, so PHASE8's [PENDING] 3b comes first and fusion is settled on our own data after an e…5e0a14c
-
docs: Phase 7 plan — data freshness and free-tier capacity
siddhm11 +1: Claude Opus 5Consolidates the measurements taken against the live deployment on 2026-07-29/30 into a plan, and logs two decisions in the doc 06 changelog per CLAUDE.md 6.2. Findings: corpus newest publication 2025-05 (today 2026-07) missing from live arXiv 290,672 unique papers across 43 categories abstracts at the 500 cap 1,437,656 (90.0%), mean live abstract 1,244 chars measured per-point cost ~2,574 bytes (2048 vector + 128 payload + 128 BQ + ~270 graph) Two things the plan deliberately blocks on: 1. A disk-usage discrepancy. This document computes ~4.1 GB against a 4 GB free tier; the console reportedly shows 400-500 MB. The latter cannot be on-disk total, since 1,596,587 float16 vectors are 3.27 GB uncompressed and are not compressed on disk -- but it matches memory_resident_bytes (529,870,848) almost exactly, suggesting the panel reports RAM. The two branches lead to opposite actions, so nothing is deleted until it is settle…9495dc3
-
feat: Cross-encoder search reranking with MiniLM-L-6
siddhm11- Add cross-encoder reranker service (app/reranker_bge_svc.py) using cross-encoder/ms-marco-MiniLM-L-6-v2 (22M params, ~200-400ms on CPU) - Integrate reranking as Step 4.5 in hybrid search pipeline (after RRF fusion, before title-match boost) for top-10 candidates - Eagerly load reranker model at startup with graceful fallback - Display BGE Rerank timing in search results UI - Fix Unicode charmap encoding crashes on Windows console (replace emoji/em-dash with ASCII in all print statements) - Fix pre-existing test bug: update search router mocks to return (ids, meta_dict) tuples matching return_meta=True contract - Fix UnboundLocalError from shadowed asyncio import in search function - Add unit tests for reranker service (3 tests) - Update CLAUDE.md and docs with architectural decision 207 tests pass, 0 failures.6ae50db
-
Phase 4 complete + Phase 4.5 instrumentation foundation
siddhm11Phase 4 (Recommendation Pipeline Fixes) - all implemented: - 4.1: Importance-weighted quota fusion (fusion.py, 20 tests) - 4.2: Turso metadata (done in Phase 3.5) - 4.3: Hungarian matching for cluster stability (10 tests) - 4.4: Category-level negative suppression (8 tests) Phase 4.5 (Instrumentation Foundation) - NEW: - Added ranker_version, candidate_source, cluster_id to interactions table - ALTER TABLE migration for existing DBs (idempotent) - Pipeline tagging: Tier 1 papers tagged by cluster/exploration - End-to-end flow: recommendations.py -> templates -> events.py -> db.py - 5 dedicated instrumentation tests TASK-TRACKER: Phase 4 marked COMPLETE, Phase 4.5 added, Phase 8 expanded Test count: 123 -> 176 (175 passing, 1 pre-existing flaky)61d5f0d
-
Phase 3 complete: Hybrid Semantic Search pipeline
siddhm11- BGE-M3 encoding (dense + sparse) via embed_svc.py - Qdrant dense search + Zilliz sparse search (parallel) - Groq LLM query rewriter with academic heuristic bypass - RRF fusion (K=60) + recency reranking (0.80/0.20) - Graceful degradation: each service can fail independently - arXiv API fallback when hybrid search returns nothing - Dockerfile for HF Spaces deployment (Docker SDK, port 7860) - 123 tests passing (88 original + 35 new, zero regressions) - All secrets via env vars only (.env.example provided)d5a6f3e