Spaces:
Running
Running
Commit History
8 days ago
-
docs: bring the current state, the data procedure and the plan to 2026-10-21
siddhm11 +1: Claude Opus 5.5- CURRENT-STATE: what the Space runs (36c792e), corpus and citation numbers after the catch-up, the daily job, which tables reset on every deploy, local text fetched again, the paper page's back link. - PHASE7 §8.1: the daily job's stages, inputs, the four catch-up runs, and the rules learned (throughput, one run at a time, schedule delay, capacity runway on cluster 2). - Redesign plan: build-order statuses for all ten PRs, a dated status note under the next steps, and traps T19-T23 for the next agent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>389e4d9
14 days ago
-
docs: record the search reranker swap and its correction in doc 06
siddhm11 +1: Claude Opus 5.5Doc 06 changelog: 2026-10-14 (mxbai-rerank-xsmall, load-aware depth, filler-free FTS, split pools) and 2026-10-15 (gain re-measured on production's truncated abstracts: +7.7%, not +10-12%; idle depth back to 10; model pins; io pool cap). CLAUDE.md section 3.4 and the quick reference, PHASE8 stage table, README and CURRENT-STATE updated to match. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>7550b06
26 days ago
-
docs: record the Llama 3.3 retirement and the Groq model chain
siddhm11 +1: Claude Opus 5.5- Doc 06 changelog: why the model is an ordered, monitored chain, why a Preview model can lead it, and the measured order. - CURRENT-STATE: a contract row for the LLM features, with the failure path and the health signal. - PHASE8: stage 1 timing now notes the model and the truncation rule. The 161-329 ms figure was measured on the retired Llama. The changelog also records a pre-existing bug found while testing. _looks_academic counts every word as an acronym under IGNORECASE, so a query of more than six words is never rewritten. It is noted, not fixed: changing it changes search behaviour and Groq usage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>7276ed9
29 days ago
-
docs(phase7): record the executed ingest and the refresh procedure
siddhm11 +1: Claude Opus 5.5Status banner (the recent shard exists; citations refreshed) and a new §8: the ordered refresh steps with the command and the check each must pass, as run on 2026-10-01. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>b5a436b
-
docs: mark phase plans and handoffs as historical records
siddhm11 +1: Claude Opus 5.5Point readers of old phase plans, handoffs and the task tracker at the current technical contract, and add a current-status addendum to the Phase 8 pipeline design. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>7a4dfe5
-
refactor: move the live benchmark under scripts
siddhm114099c75
-
docs: correct the Space metadata and the drift between the docs and production
siddhm11 +1: Claude Opus 5Space README frontmatter had five keys and no short_description, so the Space card showed no caption to anyone finding the project. Adds short_description plus explicit models/datasets links, and records why app_port, storage and license are deliberately absent (app_port defaults to 7860; persistent storage no longer exists and `suggested_storage` "will be ignored" per the HF docs; there is no LICENSE file yet, so there is nothing to declare). CLAUDE.md described one Qdrant collection, arxiv_bgem3_dense. Production runs three sharded collections with fanout, verified against /healthz/shards. Correcting this matters more than a typo: the rulebook is what every future session reads first. PHASE8 stated SEARCH_RERANK_TOP_N = 50 as current and called a window of 10 "worse than useless" -- superseded by 082e383 six days later. The rule is marked superseded with the measured sweep, plus the part the original reading missed: hybrid_search_svc truncates with fused[:top_n] and ARXIV_MAX_RESUL…a4e5676
-
docs: exact search and recommendation pipeline specification
siddhm11 +1: Claude Opus 5Consolidates both pipelines into one reference: every stage, every constant, measured latency, and whether each piece is live or pending. Records the rules that must not change and why each has already been violated once: * RRF for search, quota for recommendations. Search fuses many retrievers over one query; recommendations fuse many queries for one user. Using RRF on the recommendation side lets the dominant cluster win on rank alone and reintroduces the interest collapse the product exists to prevent. * The rerank window must exceed the result count. At SEARCH_RERANK_TOP_N=10 the cross-encoder re-ordered exactly the set already being returned. * Rescore stays on. rescore=False is no faster (615ms vs 608ms) and drops recall@10 from 100% to 57%. * Suppression on arXiv codes, not primary_topic's ~15 coarse buckets. * Title boost uses the original query, never the rewrite. * Stage timings no longer sum -- groq_time_ms overlaps encode_time_ms. Also doc…ae8d54a -
docs: Phase 7 plan — data freshness and free-tier capacity
siddhm11 +1: Claude Opus 5Consolidates the measurements taken against the live deployment on 2026-07-29/30 into a plan, and logs two decisions in the doc 06 changelog per CLAUDE.md 6.2. Findings: corpus newest publication 2025-05 (today 2026-07) missing from live arXiv 290,672 unique papers across 43 categories abstracts at the 500 cap 1,437,656 (90.0%), mean live abstract 1,244 chars measured per-point cost ~2,574 bytes (2048 vector + 128 payload + 128 BQ + ~270 graph) Two things the plan deliberately blocks on: 1. A disk-usage discrepancy. This document computes ~4.1 GB against a 4 GB free tier; the console reportedly shows 400-500 MB. The latter cannot be on-disk total, since 1,596,587 float16 vectors are 3.27 GB uncompressed and are not compressed on disk -- but it matches memory_resident_bytes (529,870,848) almost exactly, suggesting the panel reports RAM. The two branches lead to opposite actions, so nothing is deleted until it is settle…9495dc3
-
Phase 6.5 Day 1: Real Qdrant cosine scores (A1) + verification timestamp (A2)
siddhm11A1 — Feature 0 fix (qdrant_cosine_score): - recommendations.py: Switch search_by_vector() -> search_by_vector_with_scores() in both per-cluster and short-term supplement paths - Build qdrant_score_map from actual cosine scores in same pass as paper_cluster_map - Delete fake rank-decay approximation (1.0 - rank * 0.01) - Feature 0 now receives real cosines instead of synthetic linear sequence A2 — Live verification: - PHASE6-Reranker-Framing.md: Tick all 6.3 checklist items, add verification timestamp (2026-05-03: model_loaded=true, n_trees=141, fallback_active=false) Test fix: - test_integration.py: Update quota pipeline test mock from search_by_vector to search_by_vector_with_scores (returns list[dict] not list[str]) Tests: 203 passed, 0 failures3f58d41
-
Phase 6.1+6.2+6.3: Wire all 37 features into caller + health endpoint
siddhm11Phase 6.1+6.2 (feature wiring): - recommendations.py: Replace legacy 6-arg rerank_candidates() with full 37-feature call - Per-candidate cluster_importance (N,) and cluster_medoid (N,1024) from paper_cluster_map - Pre-compute is_suppressed_category and onboarding_category_match arrays - Pass qdrant_scores, user_total_saves, user_total_dismissals to reranker - Move suppression loading BEFORE rerank call (now a model feature + safety net filter) reranker.py: - compute_features() accepts per-candidate arrays (np.ndarray) for slots 23-26 - Slot 24 supports both broadcast (1D) and per-candidate (2D) medoid paths - Add is_model_loaded(), get_loaded_model_path(), get_num_trees() accessors - Add per-request feature activation logging Phase 6.3 (deployment verification): - New /healthz/reranker endpoint (app/routers/health.py) - Reports model_loaded, n_trees, feature_schema_hash, fallback_active - Wired into main.py router registration Infrastructure: - qdrant_svc.py: Add search_by_vector_wi…5856b85 -
Phase 6: LightGBM reranker integration (37-feature schema, 141-tree model)
siddhm11- Rewrote app/recommend/reranker.py: 5-feature heuristic -> 37-feature LightGBM LambdaRank with permanent heuristic fallback - Added lightgbm>=4.0,<5.0 to requirements.txt - Added production model (reranker_v1.txt, 974KB, trained on 242K citation edges) - Model repo: models/reranker-phase6/ (cloned from HF siddhm11/researchit-reranker-phase6) - Training pipeline: 3 scripts (fetch citations, generate triples, train LightGBM) - Integration tests: 7/7 passing (smoke, features, E2E, latency 0.14ms, backward compat) - Fixed 3 tests in test_reranker_diversity.py (5->37 feature schema update) - Full pytest suite: 121/121 passing, 0 regressions - Complete documentation: PHASE6-HANDOFF.md, README.md with full infra schemas - Fixed corrupted .gitignore, removed nested .git from model dir - ML Intern conversation logs preserved in docs/ML Intern docs/63b37fd
-
Phase 5: Cold-Start Onboarding + UI Redesign
siddhm11NEW FEATURES: - 2-step onboarding wizard (category select → seed paper search) - 20 curated arXiv category groups with friendly names + icons - Dark mode default with theme toggle (localStorage persistence) - Glassmorphism navbar, Inter font, brand rename to ResearchIT - Tier 0 popularity fallback: trending papers filtered by categories - Auto-mark existing users as onboarded (no disruption) NEW FILES: - app/routers/onboarding.py (5 endpoints: page, save, search, complete, skip) - app/templates/onboarding.html (standalone wizard, no base.html) - app/templates/partials/seed_search.html (embedded search for seeds) - app/static/styles.css (animations, hover effects, category badges) - docs/phases/PHASE5-Cold-Start-Onboarding-And-UI.md (design doc) - tests/test_onboarding.py (11 tests) MODIFIED: - base.html: dark theme, glassmorphism, theme toggle, Inter font - config.py: CATEGORY_GROUPS taxonomy, expand_category_groups() - db.py: user_onboarding table, 4 new functions - main.py: onboard…7106701 -
Phase 4 complete + Phase 4.5 instrumentation foundation
siddhm11Phase 4 (Recommendation Pipeline Fixes) - all implemented: - 4.1: Importance-weighted quota fusion (fusion.py, 20 tests) - 4.2: Turso metadata (done in Phase 3.5) - 4.3: Hungarian matching for cluster stability (10 tests) - 4.4: Category-level negative suppression (8 tests) Phase 4.5 (Instrumentation Foundation) - NEW: - Added ranker_version, candidate_source, cluster_id to interactions table - ALTER TABLE migration for existing DBs (idempotent) - Pipeline tagging: Tier 1 papers tagged by cluster/exploration - End-to-end flow: recommendations.py -> templates -> events.py -> db.py - 5 dedicated instrumentation tests TASK-TRACKER: Phase 4 marked COMPLETE, Phase 4.5 added, Phase 8 expanded Test count: 123 -> 176 (175 passing, 1 pre-existing flaky)61d5f0d
-
Phase 3 complete: Hybrid Semantic Search pipeline
siddhm11- BGE-M3 encoding (dense + sparse) via embed_svc.py - Qdrant dense search + Zilliz sparse search (parallel) - Groq LLM query rewriter with academic heuristic bypass - RRF fusion (K=60) + recency reranking (0.80/0.20) - Graceful degradation: each service can fail independently - arXiv API fallback when hybrid search returns nothing - Dockerfile for HF Spaces deployment (Docker SDK, port 7860) - 123 tests passing (88 original + 35 new, zero regressions) - All secrets via env vars only (.env.example provided)d5a6f3e