Spaces:
Sleeping
Sleeping
Commit History
-
Phase 3: Spark Structured Streaming -> Cassandra + Qdrant (containerized)
- streaming/spark_job.py: reads the Kafka topic and in foreachBatch embeds the micro-batch once (shared embed.py), writes docs+metadata to Cassandra (plain cassandra-driver, concurrent inserts) and upserts vectors to Qdrant. Driver- side singletons for the model, Cassandra session, and Qdrant client. Trigger AvailableNow drains the backlog then exits; at-least-once, idempotent on id. - Dockerfile.spark: python:3.11-slim + JRE + pip pyspark 3.5.1 (the official Spark image's Python 3.8 conflicts with our pinned deps); Kafka connector (spark-sql-kafka-0-10_2.12:3.5.1) pre-resolved into the Ivy cache at build. - docker-compose.yml: add qdrant (host 6533 to avoid clashes), cassandra 5.0, and the spark-job service under a "stream" profile. - config.py: CASSANDRA_HOST/PORT. - search.py: render title/text from the Qdrant payload when BM25 metadata isn't hydrated, so search works against a bare proof/cloud Qdrant. - __init__.py: slimmed so store-only consumers don't pull the sea…4519cf7
-
Phase 2: Kafka (Redpanda) broker + corpus producer
- docker-compose.yml: single-broker Redpanda (Kafka-API compatible) with host listener on :19092 and a Redpanda Console UI on :8080; healthcheck gated - producer/produce.py: streams corpus rows as id-keyed JSON events into the `documents` topic at a configurable rate; idempotent topic creation - config.py: KAFKA_BOOTSTRAP setting - tests/test_producer.py: serialization round-trip, unicode, load limit Verified: `docker compose up` brings the broker up healthy; producing 300 docs yields high-watermark 300 on the topic (confirmed via rpk). 16 tests pass.0a602d6
-
Phase 1: demo tier — FastAPI + Streamlit over a baked index, Docker-deployable
- config.py: env-driven store selection (embedded on-disk Qdrant by default, Qdrant Cloud when QDRANT_URL is set — the Option B switch) - scripts/build_index.py: bakes the demo index (on-disk Qdrant + BM25 export) from the sample corpus; search.py gains export_lexical/hydrate - app/service.py: loads the baked index (API owns the single-writer Qdrant) - app/main.py: FastAPI /search, /ingest, /health (lifespan warmup) - app/ui.py: Streamlit search + interactive ingest + arch diagram + video slot - app/dev.py: launches API + UI together (local and in-container) - Dockerfile (repo root, HF Spaces uid-1000): bakes index, warms model, one port - DEPLOY.md: Hugging Face Spaces deployment guide - tests/test_app.py: /search + ingest-then-searchable over a tiny built index Verified locally: 13 tests pass; ingest raises doc count 3000->3001 and the pasted doc ranks #1 for its unique phrase; API + UI both serve.a22b195 -
Phase 0: shared search core + reproducible corpus
Scaffold the StreamSearch repo and build the reusable heart with no infra: - embed.py: singleton MiniLM (all-MiniLM-L6-v2, 384-d), batched embed_texts - search.py: hybrid BM25 + dense (Qdrant) fused with RRF (k=60) - schema.py: Document contract + Cassandra/Qdrant definitions - data/corpus_sample.jsonl: 3000 arXiv abstracts (HF mirror, reproducible) - tests: RRF, embedding determinism, single model-load, hybrid ranking (10 pass) - pinned requirements.txt (Python 3.11), Makefile, MIT LICENSE, README Deployment: unified live site over managed free tiers (Qdrant Cloud, Astra Cassandra, Confluent/Redpanda Kafka); Spark shown via local compose + video.61d5e4e