- New Step 04.5 bm25_prefilter: cheap lexical BM25 pre-cut right after
hard_filter, before potion_broad's expensive embedding pass. Shrinks
potion's input from ~74K to 10K rows using JD-keyword overlap alone.
composite_narrow still runs its full potion+soft+bm25 blend later, once
those signals exist, so ranking quality is unchanged. Falls back to a
full passthrough on an empty/failed JD spec rather than an arbitrary cut.
- broad_retrieval/composite_narrow/precision all use fixed top_n counts
now (500/200/125) instead of a mix of top_pct and top_n.
- Moved the candidates Parquet cache down into load_candidates() itself
(was one layer up in run_preprocess, which Streamlit's per-phase
explorer bypasses). Now both the CLI and Streamlit share one cache —
verified Streamlit's eager warm-up dropped from 14.6s to 0.27s.
- Streamlit's _load_phase_input switched to st.cache_resource (avoids
copying the 100K-row frame on every cache hit; safe since every
downstream Polars st…
Add retrieval & soft/composite ranking stages
Hemprasad Badgujar
08c3203
Introduce broad retrieval (potion Model2Vec) + soft_filter scoring + composite_narrow (BM25 + z-blend) pipeline steps. Adds retrieval implementations (potion static embedder, BM25 tokenizer/scorer), ranking modules (soft_filter, composite_narrow), and tests for bm25, potion, composite, soft_filter and hard_filter behavior. Update config/settings to expose new tuning blocks (broad_retrieval, soft_filter, composite_narrow) and default embedder, refine hard_filter inactive logic and integrity toggles, and export new rank/retrieve APIs. Streamlit explorer updated to include rank stages and show JD spec; pyproject dependencies added model2vec and numpy. These changes narrow the candidate pool deterministically before expensive embedding and blend semantic + lexical + quality signals for the top-N cut.
