lumen-rag / docs /architecture.md
WickTech's picture
Deploy from push.sh
4f8feb6 verified
|
Raw History Blame Contribute Delete
12.2 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

Architecture

Components, data flow, storage backends, the async API and tracing. Back to the README.

πŸ—οΈ Architecture

flowchart TD
    subgraph Ingest
        F[Files β€” txt/md/html/pdf/docx] --> LD[Document loaders]
        LD --> C[Sentence-aware chunker]
        C --> E1[Embedder]
        E1 --> VS[(Vector store)]
        C --> BM25[(BM25 index)]
    end
    subgraph Query
        Q[Question] --> E2[Embedder]
        E2 --> DS[Cosine search]
        VS --> DS
        Q --> SS[BM25 search]
        BM25 --> SS
        DS --> RRF[Reciprocal Rank Fusion]
        SS --> RRF
        RRF --> L[LLM answerer]
        L --> A[Answer + citations]
    end
    subgraph Evaluate
        QS[Labelled Q&A set] --> H[Eval harness]
        H --> RRF
        H --> M[recall Β· precision Β· MRR Β· nDCG Β· hit-rate]
        M --> G[Regression guard β€” CI fails if scores drop]
    end
lumen_rag/
β”œβ”€β”€ ingestion/
β”‚   β”œβ”€β”€ loaders.py      txt / md / html / pdf / docx β†’ {"id","text","metadata"} (PDF title kept)
β”‚   β”œβ”€β”€ chunker.py      sentence-aware chunking with overlap
β”‚   β”œβ”€β”€ context.py      contextual chunk headers: "Document: <title>. Section: <headings>."
β”‚   └── pipeline.py     docs β†’ chunks β†’ embeddings β†’ store
β”œβ”€β”€ store.py            Store interface + persistable in-memory cosine store (.lumen_index/)
β”œβ”€β”€ pgvector_store.py   Postgres + pgvector store with namespaces
β”œβ”€β”€ embeddings.py       HashingEmbedder (offline) Β· FastEmbedEmbedder (local bge-small) Β· OpenAIEmbedder
β”œβ”€β”€ retrieval/
β”‚   β”œβ”€β”€ bm25.py         BM25Index + reciprocal_rank_fusion()
β”‚   β”œβ”€β”€ rerank.py       cross-encoder reranker (ONNX via fastembed)
β”‚   β”œβ”€β”€ decompose.py    rule-based query decomposition (opt-in, measured no gain)
β”‚   └── retriever.py    Retriever(mode=vector|bm25|hybrid), retrieve_with_confidence()
β”œβ”€β”€ reader.py           extractive SQuAD 2.0 reader (ONNX): answerability gate
β”œβ”€β”€ engine.py           RagEngine: retrieve β†’ abstention gates β†’ answer
β”œβ”€β”€ llm.py              grounded answerer with citations; offline extractive fallback
β”œβ”€β”€ eval/
β”‚   β”œβ”€β”€ metrics.py      recall Β· precision Β· MRR Β· nDCG Β· hit-rate
β”‚   β”œβ”€β”€ harness.py      labelled-set evaluation, per-category breakdowns
β”‚   β”œβ”€β”€ abstention.py   confidence sweeps, AUROC, dev/test threshold calibration
β”‚   β”œβ”€β”€ claims.py       claim-level diagnostics (retrieval vs generation failures)
β”‚   β”œβ”€β”€ evidence.py     evidence-span grading for retrieval inside one document
β”‚   └── answer_quality.py  DeepEval judging (needs an API key)
β”œβ”€β”€ tracing.py          optional OpenTelemetry spans
β”œβ”€β”€ api/                FastAPI: /ingest /ingest/upload /query /query/stream /eval /health /stats
β”‚   └── static/         interactive demo UI, served at /
└── cli.py              lumen ingest | ask | eval | eval-answers | serve
scripts/
β”œβ”€β”€ benchmark.py        v1/v2 benchmarks: --embedder --rerank --abstain --reader --contextual --decompose
β”œβ”€β”€ real_pdf_check.py   end-to-end and retrieval checks on the NIST AI RMF PDF
β”œβ”€β”€ check_eval.py       eval regression guard (enforces score thresholds)
β”œβ”€β”€ eval_report.py      PR bot: v2 metrics for head and base, markdown diff, fails on regression
β”œβ”€β”€ validate_eval_set.py  eval set consistency checks
└── loadtest.py         blocking vs non-blocking API load test

✨ Features

  • Sentence-aware chunking with configurable size + overlap.
  • Exact cosine vector search in a tiny, persistable store (swap for pgvector/Qdrant without touching callers).
  • BM25 + Reciprocal Rank Fusion β€” true sparse/dense hybrid retrieval; choose vector, bm25, or hybrid mode per query.
  • Cited answers β€” every response carries numbered source citations.
  • Streaming answers β€” Server-Sent Events on POST /query/stream.
  • Multi-format ingestion β€” .txt, .md, .html/.htm (no extra deps), .pdf (pip install lumen-rag[pdf]), .docx (pip install lumen-rag[docx]).
  • Interactive demo UI β€” upload docs, ask questions, and watch eval metrics render live at / (served by the FastAPI app, no separate frontend build).
  • Pluggable embedders β€” LUMEN_EMBEDDER=fastembed runs BAAI/bge-small-en-v1.5 locally (ONNX, CPU, no torch, pip install lumen-rag[embed]); openai uses the hosted API; hashing is the dependency-free default that keeps CI offline.
  • Abstention β€” set LUMEN_ABSTAIN_THRESHOLD and the engine refuses questions whose retrieval confidence is too low, without calling the model; responses carry abstained and confidence. Calibrated on held-out questions; see Abstention.
  • Cross-encoder reranking β€” optional second stage (LUMEN_RERANKER=BAAI/bge-reranker-base, pip install lumen-rag[rerank]): any mode over-fetches k Γ— LUMEN_RERANK_POOL candidates and a BGE cross-encoder (ONNX via fastembed, no torch) picks the final top-k. On the bundled eval set at k=3: MRR 0.974 β†’ 1.000, nDCG 0.968 β†’ 0.980.
  • Layout-aware parsing β€” optional Docling backend (LUMEN_DOCLING=1, pip install lumen-rag[docling]) for PDF/DOCX/PPTX/HTML: reading order, multi-column pages, and tables exported as Markdown.
  • πŸ“Š Evaluation harness β€” recall@k, precision@k, MRR, nDCG@k, hit-rate over a labelled question set.
  • Claim-level diagnostics β€” lumen eval-answers --claims labels every answer claim supported / distractor / unsupported and attributes wrong answers to retrieval or generation, offline.
  • Answer-quality eval β€” lumen eval-answers data/eval.jsonl judges generated answers with DeepEval (faithfulness, answer relevancy, contextual relevancy; plus contextual precision/recall for cases with an expected_answer). Needs OPENAI_API_KEY and pip install lumen-rag[answer-eval].
  • Answerability reader β€” LUMEN_READER: an extractive SQuAD 2.0 model checks a retrieved passage actually contains an answer and refuses near misses that similarity lets through; see abstention.
  • Contextual chunk headers β€” LUMEN_CONTEXTUAL=1: document title and section path embedded and indexed with every chunk; see benchmarks.
  • Query decomposition β€” LUMEN_DECOMPOSE=1: splits compound questions; measured as no gain, kept opt-in.
  • Eval regression guard β€” scripts/check_eval.py enforces minimum score thresholds in CI; the build fails if retrieval quality drops.
  • Three interfaces β€” Python API, Typer CLI, and a FastAPI server.

πŸ—„οΈ Storage backends

The engine talks to a small Store interface (add, search, chunks, save), so the vector index is swappable without touching retrieval or ingestion.

Backend Select with Persistence Notes
in-memory (default) LUMEN_STORE=memory files in LUMEN_INDEX_DIR exact cosine search, zero dependencies
Postgres + pgvector LUMEN_STORE=pgvector the database pip install 'lumen-rag[pgvector]', needs LUMEN_PG_DSN
docker compose up -d db                       # Postgres 16 with pgvector (dev credentials)
pip install -e ".[pgvector]"
export LUMEN_STORE=pgvector LUMEN_PG_DSN=postgresql://lumen:lumen@localhost:5432/lumen
lumen ingest data/eval_v2/docs                # rows are committed as they are added
lumen eval data/eval_v2/eval.jsonl            # same metrics as the in-memory store
  • Namespaces. LUMEN_NAMESPACE (default default) scopes every read and write. Several namespaces share one table and never see each other's chunks, so one deployment can serve several users or teams. PgVectorStore.clear() deletes only the current namespace.
  • Exact by default. pgvector scans exactly, so results match the in-memory store (the test suite checks the v2 metrics agree within 0.005). index="hnsw" adds an approximate index for large collections; it can change results and interacts with the namespace filter.
  • Re-ingesting a chunk id replaces it in Postgres; the in-memory store would keep both copies.
  • BM25 still runs in the app: hybrid and BM25 modes load the namespace's chunks to build the sparse index, which is fine for tens of thousands of chunks. Moving BM25 into Postgres full-text search is the next step for larger collections.
  • Qdrant and Pinecone are not implemented. The interface is small enough that an adapter is one file, but nothing beyond pgvector has been written or tested.
  • The database tests (pytest -m pgvector) need LUMEN_TEST_PG_DSN; CI runs them against a pgvector/pgvector:pg16 service container and fails if any are skipped.

⚑ Async API and tracing

The API no longer blocks its event loop. /query and /query/stream await the model on a shared, connection-pooled async client, and the blocking parts (retrieval, which can be CPU work or a database / embedding-API round trip; upload parsing; indexing) run in worker threads. Before this, /query/stream iterated a synchronous SDK stream inside an async def, so with an API key one slow model call froze every other request, /health included, and answer() built a new OpenAI client (and TLS handshake) per request.

python scripts/loadtest.py measures it against the real app with a simulated 100 ms model and offline retrieval (so it shows how the server copes while waiting, not how fast any real model is):

Variant requests/s p50 latency p95 latency loop lag max
non-blocking model call 153.5 119 ms 126 ms 28 ms
blocking model call in the loop 9.8 1125 ms 2026 ms 2033 ms

60 requests, 20 in flight, one run on a laptop; treat the ratio as indicative. "Loop lag" is the longest time a 10 ms heartbeat task could not run: it is what a blocking call costs every other client.

Tracing (OpenTelemetry, optional)

pip install 'lumen-rag[tracing]'
LUMEN_TRACING=console lumen serve                      # print spans to stdout
LUMEN_TRACING=otlp OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 lumen serve

otlp speaks OTLP/HTTP, which Langfuse, Arize Phoenix, Jaeger and Grafana Tempo accept; point it at yours with the standard OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_HEADERS / OTEL_SERVICE_NAME variables. If your application already installs a global OpenTelemetry provider, the spans appear without setting LUMEN_TRACING. Without the packages or a provider, tracing costs a couple of function calls and records nothing.

Span Attributes
lumen.query lumen.mode, lumen.k, lumen.question.chars, lumen.chunks
lumen.retrieve (child) lumen.mode, lumen.k, lumen.reranker, lumen.store, lumen.results
lumen.answer (child) lumen.offline, lumen.context_chunks, lumen.answer.chars, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens

A real run (LUMEN_TRACING=console, offline, BM25):

lumen.retrieve  parent=lumen.query  {"lumen.mode": "bm25", "lumen.k": 1, "lumen.reranker": false, "lumen.store": "VectorStore", "lumen.results": 1}
lumen.answer    parent=lumen.query  {"lumen.offline": true, "lumen.context_chunks": 1, "lumen.answer.chars": 91}
lumen.query     (root)              {"lumen.k": 1, "lumen.mode": "bm25", "lumen.question.chars": 29, "lumen.chunks": 1}

Privacy: only lengths are recorded. Question and answer text appear on spans only with LUMEN_TRACE_CONTENT=1, because other people's documents flow through this engine and you may not want them in your trace backend.

What is and is not verified: the span tree, attributes, error recording, streaming spans (including a client that disconnects mid-stream) and the OTLP wire path (spans posted as protobuf to /v1/traces on a local stub collector) are covered by tests. It has not been run against a hosted Langfuse or Phoenix instance, and token usage is only recorded for non-streaming answers.