Spaces:
Running
Running
Commit History
28 days ago
-
docs: mark phase plans and handoffs as historical records
siddhm11 +1: Claude Opus 5.5Point readers of old phase plans, handoffs and the task tracker at the current technical contract, and add a current-status addendum to the Phase 8 pipeline design. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>7a4dfe5
-
Phase 6.5: Pipeline telemetry, search UX fixes, latency profiling
siddhm11- Instrumented search pipeline: Groq rewrite, BGE-M3 encode, Qdrant+Zilliz retrieval, RRF fusion, title rerank with per-stage timing - Instrumented recommendation pipeline: clustering, ANN retrieval, metadata fetch, LightGBM rerank, MMR diversity - Split Title+Citation Rerank into Turso fetch vs compute time (exposed hidden 1.5s network call) - Added search loading overlay with pipeline stage labels - Fixed HTMX search: recommendations now hide when search starts - Fixed paper card: truncate authors (max 3 + et al), hard-truncate abstract to 500 chars - Show Groq rewrite status (skipped/rewritten/error) in both banner and breakdown - Added Groq heuristic visibility: shows skip reason (query too short, looks academic) - Added parallel task count to retrieval breakdown - New evaluation and diagnostic scripts - Removed deprecated s2_svc.pyec67b2f
-
Phase 6: LightGBM reranker integration (37-feature schema, 141-tree model)
siddhm11- Rewrote app/recommend/reranker.py: 5-feature heuristic -> 37-feature LightGBM LambdaRank with permanent heuristic fallback - Added lightgbm>=4.0,<5.0 to requirements.txt - Added production model (reranker_v1.txt, 974KB, trained on 242K citation edges) - Model repo: models/reranker-phase6/ (cloned from HF siddhm11/researchit-reranker-phase6) - Training pipeline: 3 scripts (fetch citations, generate triples, train LightGBM) - Integration tests: 7/7 passing (smoke, features, E2E, latency 0.14ms, backward compat) - Fixed 3 tests in test_reranker_diversity.py (5->37 feature schema update) - Full pytest suite: 121/121 passing, 0 regressions - Complete documentation: PHASE6-HANDOFF.md, README.md with full infra schemas - Fixed corrupted .gitignore, removed nested .git from model dir - ML Intern conversation logs preserved in docs/ML Intern docs/63b37fd
-
Phase 4 complete + Phase 4.5 instrumentation foundation
siddhm11Phase 4 (Recommendation Pipeline Fixes) - all implemented: - 4.1: Importance-weighted quota fusion (fusion.py, 20 tests) - 4.2: Turso metadata (done in Phase 3.5) - 4.3: Hungarian matching for cluster stability (10 tests) - 4.4: Category-level negative suppression (8 tests) Phase 4.5 (Instrumentation Foundation) - NEW: - Added ranker_version, candidate_source, cluster_id to interactions table - ALTER TABLE migration for existing DBs (idempotent) - Pipeline tagging: Tier 1 papers tagged by cluster/exploration - End-to-end flow: recommendations.py -> templates -> events.py -> db.py - 5 dedicated instrumentation tests TASK-TRACKER: Phase 4 marked COMPLETE, Phase 4.5 added, Phase 8 expanded Test count: 123 -> 176 (175 passing, 1 pre-existing flaky)61d5f0d
-
Phase 3 complete: Hybrid Semantic Search pipeline
siddhm11- BGE-M3 encoding (dense + sparse) via embed_svc.py - Qdrant dense search + Zilliz sparse search (parallel) - Groq LLM query rewriter with academic heuristic bypass - RRF fusion (K=60) + recency reranking (0.80/0.20) - Graceful degradation: each service can fail independently - arXiv API fallback when hybrid search returns nothing - Dockerfile for HF Spaces deployment (Docker SDK, port 7860) - 123 tests passing (88 original + 35 new, zero regressions) - All secrets via env vars only (.env.example provided)d5a6f3e