lumen-rag / docs /roadmap.md
WickTech's picture
Deploy from push.sh
4f8feb6 verified
|
Raw History Blame Contribute Delete
2.62 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

Roadmap

What is done and what is next. The reasoning and status behind it is in RAG_GAP_ANALYSIS.md. Back to the README.

πŸ—ΊοΈ Roadmap

  • BM25 hybrid search β€” sparse + dense fusion via Reciprocal Rank Fusion; vector | bm25 | hybrid mode per query
  • PDF / DOCX / HTML ingestion β€” document loaders with clean text extraction
  • Streaming answers β€” server-sent events on POST /query/stream
  • Eval regression guard β€” scripts/check_eval.py in CI; build fails if recall drops
  • PR eval report β€” sticky PR comment with metric deltas on the v2 eval set; fails on a drop over 0.01
  • pgvector adapter with namespaces behind a Store interface (Qdrant / Pinecone still open)
  • Qdrant / Pinecone adapters β€” swap the in-memory store for a production vector DB without touching engine callers
  • Cross-encoder reranker β€” second-stage BGE reranking via fastembed (LUMEN_RERANKER)
  • Layout-aware ingestion β€” Docling backend for PDF/DOCX/PPTX/HTML (LUMEN_DOCLING=1)
  • Answer-quality eval β€” DeepEval faithfulness / relevancy metrics (lumen eval-answers)
  • Claim-level diagnostics β€” lumen eval-answers --claims attributes wrong answers to retrieval vs generation, offline
  • Multi-tenant namespaces β€” LUMEN_NAMESPACE isolates collections per user/team (pgvector backend)
  • Async FastAPI endpoints β€” non-blocking model calls on a pooled async client, blocking work in worker threads (async DB and embedder clients are still open)
  • Real embedder benchmark β€” local bge-small (fastembed) next to the hashing baseline; OpenAI row pending
  • Abstention β€” confidence-gated refusal, calibrated on held-out questions, measured end to end
  • Answerability check beyond similarity β€” an extractive SQuAD 2.0 reader refuses near misses that similarity lets through
  • Query decomposition for multi-hop questions: measured, no gain with a real embedder (see benchmarks); kept opt-in
  • LLM query rewriting for implicit multi-hop questions (needs an API key with credit)
  • Contextual chunk headers β€” offline title + section headers before embedding and BM25; helps keyword search, small effect on dense (see benchmarks)
  • LLM-written chunk context β€” Anthropic's full version, needs an API key
  • Public benchmark β€” BEIR SciFact alongside the in-repo sets
  • Fine-tuned embedding adapter β€” LoRA adapter trained on domain Q&A pairs to improve recall