Unblock the event loop during ingest, fit summarize in free-tier quota
beerohan
7935a97
Profiling a 120-page PDF showed ingest cost is almost entirely embedding:
parsing 0.09s, cleaning 0.00s, embedding 6.51s (98.6%). That embedding ran
synchronously inside the async /upload route, so it blocked the whole event
loop - concurrent uploads serialized and status polling stalled behind them,
which is what made multi-file uploads feel frozen.
Move add_document and the PDF/DOCX/text parsing onto worker threads via
asyncio.to_thread. /status now answers in ~3ms while uploads run, instead of
queueing behind a full embed. Wall time for three PDFs improves modestly
(12s -> 10s); embedding is CPU-bound, so threads cannot multiply throughput
on shared cores.
Larger embedding batches were benchmarked and rejected: batch_size 10/64/128
measured 3.47s/3.73s/3.80s, so the default of 10 stays.
Summarize retrieved similarity_top_k=8 chunks, roughly 8k input tokens, which
alone exhausts Groq's free-tier 8,000 tokens-per-minute budget and returned a
429 before generating anything. Lower it…
