Spaces:
Runtime error
Runtime error
Commit History
-
feat(p2): implement data ingestion pipeline with dense and BM25 sparse vectors
sourabh gupta- environment.yaml, packages/vantage_core/pyproject.toml: Add fastembed (BM25) and sentence-transformers - packages/vantage_core/models.py: Core Document and Chunk models - packages/vantage_core/loader.py: Stream HuffPost JSONL generator skipping noise rows - packages/vantage_core/chunker.py: RecursiveCharacterTextSplitter chunking safety net - packages/vantage_core/adapters/embedder_st.py: STEmbedder for all-MiniLM-L6-v2 dense embeddings - packages/vantage_core/adapters/vector_qdrant.py: Hybrid dual-vector (dense + BM25 sparse) upserts with Qdrant Vector Repo - scripts/ingest.py: Orchestrate ingestion batch flushes (sample 2k docs default) - scripts/peek.py: Verification helper scrolling payloads6985a83