Spaces:
Sleeping
Sleeping
Commit History
-
Phase 4: fix demo reset — --profile stream down -v wipes the Spark checkpoint
A plain 'docker compose down -v' leaves streamsearch_spark_checkpoints behind because spark-job is behind the stream profile, so the next run fails with 'offset was changed from N to M, some data may have been missed'. The reset must use --profile stream on the down. Also note: produce/run once per reset (the job is checkpointed). Verified end-to-end: reset -> produce 300 -> Spark Batch 0 wrote 300 -> Cassandra 300, Qdrant 300.de0c70d -
Phase 4: make record_demo.md commands runnable on Windows/PowerShell
Verified every shot against the live stack. Fixes: use the venv for Python commands (bare python misses confluent-kafka); PowerShell-native Qdrant count via Invoke-RestMethod (jq/curl -s unavailable); single-line cqlsh (drop bash \ continuation); add --build to first Spark run.3b68cb1
-
Phase 4: unify — query the streamed proof-tier index with the shared search stack
- scripts/query_proof.py: run the SAME HybridSearcher (BM25 + dense + RRF) over the Qdrant populated by the Spark job, hydrating BM25 metadata from Cassandra. - scripts/record_demo.md: shot list for the ~90s pipeline walkthrough video. - README: demo-video section, unify-check command, Phase 4 status.615fa4c
-
Phase 1: demo tier — FastAPI + Streamlit over a baked index, Docker-deployable
- config.py: env-driven store selection (embedded on-disk Qdrant by default, Qdrant Cloud when QDRANT_URL is set — the Option B switch) - scripts/build_index.py: bakes the demo index (on-disk Qdrant + BM25 export) from the sample corpus; search.py gains export_lexical/hydrate - app/service.py: loads the baked index (API owns the single-writer Qdrant) - app/main.py: FastAPI /search, /ingest, /health (lifespan warmup) - app/ui.py: Streamlit search + interactive ingest + arch diagram + video slot - app/dev.py: launches API + UI together (local and in-container) - Dockerfile (repo root, HF Spaces uid-1000): bakes index, warms model, one port - DEPLOY.md: Hugging Face Spaces deployment guide - tests/test_app.py: /search + ingest-then-searchable over a tiny built index Verified locally: 13 tests pass; ingest raises doc count 3000->3001 and the pasted doc ranks #1 for its unique phrase; API + UI both serve.a22b195 -
Phase 0: shared search core + reproducible corpus
Scaffold the StreamSearch repo and build the reusable heart with no infra: - embed.py: singleton MiniLM (all-MiniLM-L6-v2, 384-d), batched embed_texts - search.py: hybrid BM25 + dense (Qdrant) fused with RRF (k=60) - schema.py: Document contract + Cassandra/Qdrant definitions - data/corpus_sample.jsonl: 3000 arXiv abstracts (HF mirror, reproducible) - tests: RRF, embedding determinism, single model-load, hybrid ranking (10 pass) - pinned requirements.txt (Python 3.11), Makefile, MIT LICENSE, README Deployment: unified live site over managed free tiers (Qdrant Cloud, Astra Cassandra, Confluent/Redpanda Kafka); Spark shown via local compose + video.61d5e4e