Download docs/architecture.md from WickTech/lumen-rag: direct link, hf CLI and curl.
- Browser
- Download file 12.2 kB
-
https://huggingface.co/spaces/WickTech/lumen-rag/resolve/main/docs/architecture.md
- Command line
-
hf download hf://spaces/WickTech/lumen-rag/docs/architecture.md
-
curl -L -o architecture.md https://huggingface.co/spaces/WickTech/lumen-rag/resolve/main/docs/architecture.md
A newer version of the Gradio SDK is available: 6.32.0
Architecture
Components, data flow, storage backends, the async API and tracing. Back to the README.
ποΈ Architecture
flowchart TD
subgraph Ingest
F[Files β txt/md/html/pdf/docx] --> LD[Document loaders]
LD --> C[Sentence-aware chunker]
C --> E1[Embedder]
E1 --> VS[(Vector store)]
C --> BM25[(BM25 index)]
end
subgraph Query
Q[Question] --> E2[Embedder]
E2 --> DS[Cosine search]
VS --> DS
Q --> SS[BM25 search]
BM25 --> SS
DS --> RRF[Reciprocal Rank Fusion]
SS --> RRF
RRF --> L[LLM answerer]
L --> A[Answer + citations]
end
subgraph Evaluate
QS[Labelled Q&A set] --> H[Eval harness]
H --> RRF
H --> M[recall Β· precision Β· MRR Β· nDCG Β· hit-rate]
M --> G[Regression guard β CI fails if scores drop]
end
lumen_rag/
βββ ingestion/
β βββ loaders.py txt / md / html / pdf / docx β {"id","text","metadata"} (PDF title kept)
β βββ chunker.py sentence-aware chunking with overlap
β βββ context.py contextual chunk headers: "Document: <title>. Section: <headings>."
β βββ pipeline.py docs β chunks β embeddings β store
βββ store.py Store interface + persistable in-memory cosine store (.lumen_index/)
βββ pgvector_store.py Postgres + pgvector store with namespaces
βββ embeddings.py HashingEmbedder (offline) Β· FastEmbedEmbedder (local bge-small) Β· OpenAIEmbedder
βββ retrieval/
β βββ bm25.py BM25Index + reciprocal_rank_fusion()
β βββ rerank.py cross-encoder reranker (ONNX via fastembed)
β βββ decompose.py rule-based query decomposition (opt-in, measured no gain)
β βββ retriever.py Retriever(mode=vector|bm25|hybrid), retrieve_with_confidence()
βββ reader.py extractive SQuAD 2.0 reader (ONNX): answerability gate
βββ engine.py RagEngine: retrieve β abstention gates β answer
βββ llm.py grounded answerer with citations; offline extractive fallback
βββ eval/
β βββ metrics.py recall Β· precision Β· MRR Β· nDCG Β· hit-rate
β βββ harness.py labelled-set evaluation, per-category breakdowns
β βββ abstention.py confidence sweeps, AUROC, dev/test threshold calibration
β βββ claims.py claim-level diagnostics (retrieval vs generation failures)
β βββ evidence.py evidence-span grading for retrieval inside one document
β βββ answer_quality.py DeepEval judging (needs an API key)
βββ tracing.py optional OpenTelemetry spans
βββ api/ FastAPI: /ingest /ingest/upload /query /query/stream /eval /health /stats
β βββ static/ interactive demo UI, served at /
βββ cli.py lumen ingest | ask | eval | eval-answers | serve
scripts/
βββ benchmark.py v1/v2 benchmarks: --embedder --rerank --abstain --reader --contextual --decompose
βββ real_pdf_check.py end-to-end and retrieval checks on the NIST AI RMF PDF
βββ check_eval.py eval regression guard (enforces score thresholds)
βββ eval_report.py PR bot: v2 metrics for head and base, markdown diff, fails on regression
βββ validate_eval_set.py eval set consistency checks
βββ loadtest.py blocking vs non-blocking API load test
β¨ Features
- Sentence-aware chunking with configurable size + overlap.
- Exact cosine vector search in a tiny, persistable store (swap for pgvector/Qdrant without touching callers).
- BM25 + Reciprocal Rank Fusion β true sparse/dense hybrid retrieval; choose
vector,bm25, orhybridmode per query. - Cited answers β every response carries numbered source citations.
- Streaming answers β Server-Sent Events on
POST /query/stream. - Multi-format ingestion β
.txt,.md,.html/.htm(no extra deps),.pdf(pip install lumen-rag[pdf]),.docx(pip install lumen-rag[docx]). - Interactive demo UI β upload docs, ask questions, and watch eval metrics render live at
/(served by the FastAPI app, no separate frontend build). - Pluggable embedders β
LUMEN_EMBEDDER=fastembedrunsBAAI/bge-small-en-v1.5locally (ONNX, CPU, no torch,pip install lumen-rag[embed]);openaiuses the hosted API;hashingis the dependency-free default that keeps CI offline. - Abstention β set
LUMEN_ABSTAIN_THRESHOLDand the engine refuses questions whose retrieval confidence is too low, without calling the model; responses carryabstainedandconfidence. Calibrated on held-out questions; see Abstention. - Cross-encoder reranking β optional second stage (
LUMEN_RERANKER=BAAI/bge-reranker-base,pip install lumen-rag[rerank]): any mode over-fetchesk Γ LUMEN_RERANK_POOLcandidates and a BGE cross-encoder (ONNX via fastembed, no torch) picks the final top-k. On the bundled eval set at k=3: MRR 0.974 β 1.000, nDCG 0.968 β 0.980. - Layout-aware parsing β optional Docling backend (
LUMEN_DOCLING=1,pip install lumen-rag[docling]) for PDF/DOCX/PPTX/HTML: reading order, multi-column pages, and tables exported as Markdown. - π Evaluation harness β recall@k, precision@k, MRR, nDCG@k, hit-rate over a labelled question set.
- Claim-level diagnostics β
lumen eval-answers --claimslabels every answer claim supported / distractor / unsupported and attributes wrong answers to retrieval or generation, offline. - Answer-quality eval β
lumen eval-answers data/eval.jsonljudges generated answers with DeepEval (faithfulness, answer relevancy, contextual relevancy; plus contextual precision/recall for cases with anexpected_answer). NeedsOPENAI_API_KEYandpip install lumen-rag[answer-eval]. - Answerability reader β
LUMEN_READER: an extractive SQuAD 2.0 model checks a retrieved passage actually contains an answer and refuses near misses that similarity lets through; see abstention. - Contextual chunk headers β
LUMEN_CONTEXTUAL=1: document title and section path embedded and indexed with every chunk; see benchmarks. - Query decomposition β
LUMEN_DECOMPOSE=1: splits compound questions; measured as no gain, kept opt-in. - Eval regression guard β
scripts/check_eval.pyenforces minimum score thresholds in CI; the build fails if retrieval quality drops. - Three interfaces β Python API, Typer CLI, and a FastAPI server.
ποΈ Storage backends
The engine talks to a small Store interface (add, search, chunks, save), so the
vector index is swappable without touching retrieval or ingestion.
| Backend | Select with | Persistence | Notes |
|---|---|---|---|
| in-memory (default) | LUMEN_STORE=memory |
files in LUMEN_INDEX_DIR |
exact cosine search, zero dependencies |
| Postgres + pgvector | LUMEN_STORE=pgvector |
the database | pip install 'lumen-rag[pgvector]', needs LUMEN_PG_DSN |
docker compose up -d db # Postgres 16 with pgvector (dev credentials)
pip install -e ".[pgvector]"
export LUMEN_STORE=pgvector LUMEN_PG_DSN=postgresql://lumen:lumen@localhost:5432/lumen
lumen ingest data/eval_v2/docs # rows are committed as they are added
lumen eval data/eval_v2/eval.jsonl # same metrics as the in-memory store
- Namespaces.
LUMEN_NAMESPACE(defaultdefault) scopes every read and write. Several namespaces share one table and never see each other's chunks, so one deployment can serve several users or teams.PgVectorStore.clear()deletes only the current namespace. - Exact by default. pgvector scans exactly, so results match the in-memory store (the test
suite checks the v2 metrics agree within 0.005).
index="hnsw"adds an approximate index for large collections; it can change results and interacts with the namespace filter. - Re-ingesting a chunk id replaces it in Postgres; the in-memory store would keep both copies.
- BM25 still runs in the app: hybrid and BM25 modes load the namespace's chunks to build the sparse index, which is fine for tens of thousands of chunks. Moving BM25 into Postgres full-text search is the next step for larger collections.
- Qdrant and Pinecone are not implemented. The interface is small enough that an adapter is one file, but nothing beyond pgvector has been written or tested.
- The database tests (
pytest -m pgvector) needLUMEN_TEST_PG_DSN; CI runs them against apgvector/pgvector:pg16service container and fails if any are skipped.
β‘ Async API and tracing
The API no longer blocks its event loop. /query and /query/stream await the model on a shared,
connection-pooled async client, and the blocking parts (retrieval, which can be CPU work or a database /
embedding-API round trip; upload parsing; indexing) run in worker threads. Before this, /query/stream
iterated a synchronous SDK stream inside an async def, so with an API key one slow model call froze every
other request, /health included, and answer() built a new OpenAI client (and TLS handshake) per request.
python scripts/loadtest.py measures it against the real app with a simulated 100 ms model and offline
retrieval (so it shows how the server copes while waiting, not how fast any real model is):
| Variant | requests/s | p50 latency | p95 latency | loop lag max |
|---|---|---|---|---|
| non-blocking model call | 153.5 | 119 ms | 126 ms | 28 ms |
| blocking model call in the loop | 9.8 | 1125 ms | 2026 ms | 2033 ms |
60 requests, 20 in flight, one run on a laptop; treat the ratio as indicative. "Loop lag" is the longest time a 10 ms heartbeat task could not run: it is what a blocking call costs every other client.
Tracing (OpenTelemetry, optional)
pip install 'lumen-rag[tracing]'
LUMEN_TRACING=console lumen serve # print spans to stdout
LUMEN_TRACING=otlp OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 lumen serve
otlp speaks OTLP/HTTP, which Langfuse, Arize Phoenix, Jaeger and Grafana Tempo accept; point it at yours with the
standard OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_HEADERS / OTEL_SERVICE_NAME variables. If your
application already installs a global OpenTelemetry provider, the spans appear without setting LUMEN_TRACING.
Without the packages or a provider, tracing costs a couple of function calls and records nothing.
| Span | Attributes |
|---|---|
lumen.query |
lumen.mode, lumen.k, lumen.question.chars, lumen.chunks |
lumen.retrieve (child) |
lumen.mode, lumen.k, lumen.reranker, lumen.store, lumen.results |
lumen.answer (child) |
lumen.offline, lumen.context_chunks, lumen.answer.chars, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens |
A real run (LUMEN_TRACING=console, offline, BM25):
lumen.retrieve parent=lumen.query {"lumen.mode": "bm25", "lumen.k": 1, "lumen.reranker": false, "lumen.store": "VectorStore", "lumen.results": 1}
lumen.answer parent=lumen.query {"lumen.offline": true, "lumen.context_chunks": 1, "lumen.answer.chars": 91}
lumen.query (root) {"lumen.k": 1, "lumen.mode": "bm25", "lumen.question.chars": 29, "lumen.chunks": 1}
Privacy: only lengths are recorded. Question and answer text appear on spans only with LUMEN_TRACE_CONTENT=1,
because other people's documents flow through this engine and you may not want them in your trace backend.
What is and is not verified: the span tree, attributes, error recording, streaming spans (including a client that
disconnects mid-stream) and the OTLP wire path (spans posted as protobuf to /v1/traces on a local stub collector)
are covered by tests. It has not been run against a hosted Langfuse or Phoenix instance, and token usage is only
recorded for non-streaming answers.