Unblock the event loop during ingest, fit summarize in free-tier quota
beerohan
7935a97
Profiling a 120-page PDF showed ingest cost is almost entirely embedding:
parsing 0.09s, cleaning 0.00s, embedding 6.51s (98.6%). That embedding ran
synchronously inside the async /upload route, so it blocked the whole event
loop - concurrent uploads serialized and status polling stalled behind them,
which is what made multi-file uploads feel frozen.
Move add_document and the PDF/DOCX/text parsing onto worker threads via
asyncio.to_thread. /status now answers in ~3ms while uploads run, instead of
queueing behind a full embed. Wall time for three PDFs improves modestly
(12s -> 10s); embedding is CPU-bound, so threads cannot multiply throughput
on shared cores.
Larger embedding batches were benchmarked and rejected: batch_size 10/64/128
measured 3.47s/3.73s/3.80s, so the default of 10 stays.
Summarize retrieved similarity_top_k=8 chunks, roughly 8k input tokens, which
alone exhausts Groq's free-tier 8,000 tokens-per-minute budget and returned a
429 before generating anything. Lower it…
Rebuild frontend in React, add doc/session management, fix broken deps
beerohan
dd4a03a
Frontend rebuilt as a React + Vite + TypeScript SPA, replacing the vanilla
JS/CSS app. Adds dark mode, a document list with per-document delete,
multi-file upload, cancellable streaming, and chat history that survives a
page reload. Served by FastAPI from static/, which is now a build artifact
produced by the Node stage of a multi-stage Dockerfile.
Backend additions:
- DELETE /documents/{name} removes a single document from the Chroma index
- Chat history persisted to SQLite (app/services/history_service.py) with
GET/PATCH/DELETE /sessions routes; transcripts survive process restarts
Performance:
- /stream_query fetched citation sources by calling query() a second time,
running a full LLM generation whose text was discarded. Replaced with a
retrieval-only path, saving ~0.6s and one Groq call per message.
- Embedding model is warmed during startup so the first request no longer
pays the model-load cost.
Pre-existing breakages fixed:
- requirements.txt pinned llama-index-embed…
