Spaces:
Running
Running
Commit History
-
feat(ingest): add --encode-only mode
siddhm11 +1: Claude Opus 5Validates the BGE-M3 encode path without database credentials: fetches and encodes real papers, reports dense dimension, L2 norm and sparse term counts, and writes nothing. Lets the GPU path be proven on Kaggle before any secret reaches that platform, and before committing to a full backfill. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>45b0962 -
feat: arXiv ingestion pipeline
siddhm11 +1: Claude Opus 5The corpus is a one-time Kaggle snapshot. Measured against the live arXiv API on 2026-07-29: newest publication in corpus : 2025-05 today : 2026-07-29 Papers published since 2025-06 in eight of the configured categories: cs.AI 62,438 cs.LG 58,747 cs.CV 43,269 cs.CL 29,295 quant-ph 21,026 cs.RO 14,370 stat.ML 7,841 cs.IR 5,441 ~242k paper-slots against a 1.6M corpus. arXiv has published nearly half again as many cs.AI papers in that window as the entire index contains, so roughly a third of that field is simply absent. No ranking work fixes a paper that is not in the database. scripts/ingest_arxiv.py fetch + parse + orchestrate, checkpointed scripts/ingest_backends.py BGE-M3 encode, Qdrant/Zilliz/Turso upsert Three consistency decisions worth recording: 1. Embedding text matches the original ingest exactly -- title[:256] plus abstract[:1024], max_length=512. New vectors share an AN…ef7b024 -
perf: local metadata sidecar + fine-grained categories
siddhm11 +1: Claude Opus 5Profiled the live Space on a warm, idle container (4 novel queries): Qdrant+Zilliz 1803-2891 ms <- dominant Turso metadata 1100-1270 ms BGE-M3 encode 285-642 ms Groq rewrite 161-329 ms cross-encoder 139-182 ms rerank compute 1 ms One run happened not to reorder during title-boost, so the old buggy pop succeeded and printed the true split (Turso 1132ms / compute 1ms), confirming the mislabelling fixed in the previous commit. Metadata sidecar ---------------- Turso has a single index (arxiv_id). Point lookups are fine; everything else is a full scan of 1.6M rows -- MIN/MAX(update_date) does not complete within 100s, and trending is LIKE '%code%' plus a temp B-tree sort, which the code comments put at ~15s cold. That query runs during onboarding, so the slowest path in the app is the one brand-new users hit first. app/local_meta.py mirrors the table into a local SQLite shipped in the image, carrying the indexes Turso lacks, and sits betwe…78bab6f
-
feat: Cross-encoder search reranking with MiniLM-L-6
siddhm11- Add cross-encoder reranker service (app/reranker_bge_svc.py) using cross-encoder/ms-marco-MiniLM-L-6-v2 (22M params, ~200-400ms on CPU) - Integrate reranking as Step 4.5 in hybrid search pipeline (after RRF fusion, before title-match boost) for top-10 candidates - Eagerly load reranker model at startup with graceful fallback - Display BGE Rerank timing in search results UI - Fix Unicode charmap encoding crashes on Windows console (replace emoji/em-dash with ASCII in all print statements) - Fix pre-existing test bug: update search router mocks to return (ids, meta_dict) tuples matching return_meta=True contract - Fix UnboundLocalError from shadowed asyncio import in search function - Add unit tests for reranker service (3 tests) - Update CLAUDE.md and docs with architectural decision 207 tests pass, 0 failures.6ae50db
-
Phase 6.5: Pipeline telemetry, search UX fixes, latency profiling
siddhm11- Instrumented search pipeline: Groq rewrite, BGE-M3 encode, Qdrant+Zilliz retrieval, RRF fusion, title rerank with per-stage timing - Instrumented recommendation pipeline: clustering, ANN retrieval, metadata fetch, LightGBM rerank, MMR diversity - Split Title+Citation Rerank into Turso fetch vs compute time (exposed hidden 1.5s network call) - Added search loading overlay with pipeline stage labels - Fixed HTMX search: recommendations now hide when search starts - Fixed paper card: truncate authors (max 3 + et al), hard-truncate abstract to 500 chars - Show Groq rewrite status (skipped/rewritten/error) in both banner and breakdown - Added Groq heuristic visibility: shows skip reason (query too short, looks academic) - Added parallel task count to retrieval breakdown - New evaluation and diagnostic scripts - Removed deprecated s2_svc.pyec67b2f
-
Phase 6: LightGBM reranker integration (37-feature schema, 141-tree model)
siddhm11- Rewrote app/recommend/reranker.py: 5-feature heuristic -> 37-feature LightGBM LambdaRank with permanent heuristic fallback - Added lightgbm>=4.0,<5.0 to requirements.txt - Added production model (reranker_v1.txt, 974KB, trained on 242K citation edges) - Model repo: models/reranker-phase6/ (cloned from HF siddhm11/researchit-reranker-phase6) - Training pipeline: 3 scripts (fetch citations, generate triples, train LightGBM) - Integration tests: 7/7 passing (smoke, features, E2E, latency 0.14ms, backward compat) - Fixed 3 tests in test_reranker_diversity.py (5->37 feature schema update) - Full pytest suite: 121/121 passing, 0 regressions - Complete documentation: PHASE6-HANDOFF.md, README.md with full infra schemas - Fixed corrupted .gitignore, removed nested .git from model dir - ML Intern conversation logs preserved in docs/ML Intern docs/63b37fd
-
Phase 6: Add arxiv_ids export script (IDs file on HF model repo)
siddhm11cb4a1c8