Spaces:

amer003100
/

vectorless_RAG

Running

App Files Files Community

vectorless_RAG / README.md

AI Assistant

docs: accuracy pass on README (fix latency claims, subsystem count, units, links, and naming)

f7154a3 about 14 hours ago

preview code

Raw History Blame Contribute Delete

27 kB

metadata

title: Atlas Document Intelligence API
emoji: 🌿
colorFrom: green
colorTo: blue
sdk: docker
app_port: 7860
pinned: false

🌿 Atlas — Enterprise Vectorless RAG & Document Intelligence Platform

Understand Documents. Discover Knowledge. Reason with Exact Page Verification.
Production-grade Vectorless RAG platform powered by hierarchical outline reasoning (PageIndex + PyMuPDF), pure-Python lexical BM25 with Nucleus (top_p) Cumulative Probability Cutoff, Sub-Node Paragraph Pruning with Exact Inner-Page Citation Pinning, an Interactive Study Studio (Mind Maps + Quizzes), a Research & Study FastMCP Suite (5 zero-paid-key tools), and resilient dual LLM inference (Hugging Face + Groq zero-downtime failover) on PostgreSQL persistence.

📑 جدول المحتويات / Table of Contents

Overview & Core Philosophy
Complete System Architecture
Vectorless Hybrid Retrieval & Nucleus (top_p) Cutoff
Key Features & Capabilities
Industry Benchmarks & Quantitative Evaluation (95 Questions)
Tech Stack
API Endpoints Reference
Environment Variables
Live Deployments

🌟 Overview & Core Philosophy

Atlas redefines Document Intelligence and Retrieval-Augmented Generation (RAG) by completely eliminating vector embeddings, arbitrary chunking, and similarity drift. Instead of blindly slicing documents into 500-token chunks and computing cosine distance over a vector database, Atlas models documents as Hierarchical Semantic Trees (PageIndex + PyMuPDF).

Every section, chapter, table, and footnote preserves its parent-child lineage and exact physical page coordinates. When a user queries the system:

Hybrid Retrieval: Runs lexical BM25 search and weighted RRF fusion in ~2ms, while the LLM tree-search step adds model latency, achieving an end-to-end average query latency of 1.45s.
Nucleus Consensus (top_p = 0.75): Combines rankings via Weighted Reciprocal Rank Fusion (RRF: 0.6 Tree, 0.4 BM25) and applies temperature-scaled Softmax to dynamically isolate the minimal sufficient set of nodes (single-hop queries truncate to 1 node; multi-hop queries preserve 2 nodes).
Sub-Node Paragraph Pruning & Exact Inner-Page Pinning: Filters long sections down to high-density paragraphs and pins citations to the exact --- [PAGE X] --- where evidence resides.
Fallback & Extended Research: When out-of-document facts, arXiv papers, mathematical ratios, or YouTube lectures are needed, Atlas activates its built-in Research & Study FastMCP Suite with zero external API fees.

🏛️ Complete System Architecture

flowchart TB
    subgraph Client_Layer["🖥️ Frontend Client (React 19 + Vite / Vercel)"]
        direction TB
        UI_Sidebar["Sidebar (Sessions, Multi-Chat & Unread Indicators)"]
        UI_Explorer["Document Structure Explorer (Hierarchical Tree View)"]
        UI_Studio["Study Studio (Interactive Mind Map & Flashcard Quiz)"]
        UI_Chat["Concurrent Async Chat (Multi-Session Streaming)"]
        UI_Sources["Interactive Source Badges (📄 PDF Pages & 🌐 Web Badges)"]
        UI_Memory["Knowledge Memory & Workspace Dossier View"]
        UI_SSE["SSE Stream Parser & Background Dispatcher"]
        UI_Storage[("Browser LocalStorage Cache")]

        UI_Sidebar <--> UI_Chat
        UI_Chat <--> UI_Sources
        UI_Chat <--> UI_SSE
        UI_Chat <--> UI_Storage
        UI_Sidebar <--> UI_Studio
    end

    subgraph API_Layer["⚡ Backend Gateway (FastAPI / Hugging Face Spaces)"]
        direction TB
        Router_Chat["/ask & /chat Routers (Streaming SSE, Multi-Doc)"]
        Router_Studio["/studio Router (/mindmap, /quiz MCQs & Flashcards)"]
        Router_Tools["/tools Router (Web, arXiv, YouTube, Math)"]
        Router_Docs["/documents Router (Upload, Ingestion, Trees)"]
        Router_Memory["/memory Router (Dossiers, Workspace Synthesis)"]
        Router_MCP["FastMCP Server Mount (/mcp via SSE/HTTP)"]
    end

    subgraph FastMCP_Suite["🌐 Research & Study FastMCP Suite (Zero Paid Keys)"]
        direction LR
        Tool_Web["Deep Web Reader (httpx + BS4/Trafilatura, 6s timeout)"]
        Tool_ArXiv["Academic arXiv Tool (export.arxiv.org Atom API)"]
        Tool_YT["YouTube Transcript (Segmenter & Chapter Analyzer)"]
        Tool_Math["Safe AST Math Evaluator (AST Whitelist, Zero eval)"]
        Tool_Atlas["Atlas Internal Tools (list_docs, query, dossiers)"]
    end

    subgraph Core_Engine["🧠 Vectorless RAG & Reasoning Pipelines"]
        direction TB
        Intent_Classifier["Fast Intent Classifier (< 0.02ms Deterministic)"]
        Doc_Service["DocumentService (PageIndex API + PyMuPDF Page Markers)"]
        BM25_Engine["Zero-Dependency Okapi BM25 (Arabic/English Stemmer)"]
        RAG_Pipeline["VectorlessRAG (Weighted RRF + Nucleus top_p Softmax)"]
        Pruner["Sub-Node Paragraph Pruning & Exact Inner-Page Pinning"]
        LLM_Service["LLMService (Tree Search, Grounded Synthesis, Dual Failover)"]
        Memory_Service["MemoryService (Short-Term, Long-Term, Workspace Dossiers)"]
    end

    subgraph AI_Inference["🤖 Resilient Dual-Provider LLM Engine"]
        direction LR
        HF_LLM["Hugging Face Serverless (Primary: meta-llama/Llama-3.1-8B-Instruct)"]
        Groq_LLM["Groq Cloud API (Zero-Downtime Fallback: qwen/qwen3.8-27b)"]
        HF_LLM -.->|Auto-Failover on 402/Limit/Error| Groq_LLM
    end

    subgraph Persistence_Layer["💾 Enterprise Persistence & Database"]
        direction TB
        Mem_Store["MemoryStorage (In-Memory Hot Cache)"]

        subgraph DB_System["PostgreSQL / Supabase (AWS eu-west-1)"]
            PG_Pool["Threaded Connection Pool (psycopg2)"]
            T_Meta[("doc_meta (Metadata & Statuses)")]
            T_Indices[("doc_indices (JSONB Trees & Injected Markers)")]
            T_Chats[("chats & messages (Isolated Sessions)")]
            T_Docs[("chat_documents (Multi-Doc Associations)")]
            T_Memory[("knowledge_memory (Per-Doc Dossiers)")]
            T_Global[("global_knowledge_summary (Workspace)")]
        end

        SQLite_Fallback[("SQLite Local Fallback (memory.db)")]
    end

    %% Wiring
    Client_Layer <==>|REST API & SSE Streams| API_Layer
    API_Layer <--> Router_MCP
    Router_MCP <--> FastMCP_Suite
    Router_Studio --> LLM_Service
    Router_Chat --> Intent_Classifier
    Intent_Classifier --> RAG_Pipeline
    RAG_Pipeline --> BM25_Engine & Doc_Service
    RAG_Pipeline --> Pruner
    Pruner --> LLM_Service
    LLM_Service <==> AI_Inference
    Router_Memory --> Memory_Service
    Mem_Store <--> PG_Pool
    PG_Pool --> T_Meta & T_Indices & T_Chats & T_Docs & T_Memory & T_Global
    PG_Pool -.->|Offline Failover| SQLite_Fallback

🧭 Vectorless Hybrid Retrieval & Nucleus (`top_p`) Cutoff

Instead of relying on arbitrary static top_k=3 cutoffs that pollute LLM context on single-fact queries, Atlas dynamically sizes the retrieved node set via Temperature-Scaled Softmax Nucleus Retrieval:

flowchart TD
    Q["User Query"] --> Clf{"Fast Intent Classifier (< 0.02ms)"}

    Clf -->|"Academic Paper"| MCP_Arxiv["FastMCP arXiv Tool (export.arxiv.org)"]
    Clf -->|"Live Market / Web"| MCP_Web["FastMCP Web Reader (Clean Markdown)"]
    Clf -->|"Global / Overview"| Dossier["Pre-computed Executive Dossier + Root Chapters"]
    Clf -->|"Specific / Numerical"| Hybrid["Parallel Hybrid Search (BM25 + Fusion ~2ms)"]

    Hybrid --> Tree["Hierarchical LLM Tree Traversal (Weight 0.6)"]
    Hybrid --> BM25["Pure-Python Okapi BM25 (Weight 0.4)"]

    Tree & BM25 --> Consensus["Consensus Score: S(d) = sum w_L / (60 + rank_L) * Density"]
    Consensus --> Softmax["Temperature-Scaled Softmax (T = 0.15): P(node_i)"]
    Softmax --> Nucleus{"Accumulate P(node_i) until sum >= top_p (0.75)"}

    Nucleus -->|Single-Node Local Query| Top1["Select Minimal Sufficient Top-1 Node"]
    Nucleus -->|Multi-Hop Complex Query| Top2["Preserve High-Relevance Top-2 Nodes (min_k=2)"]

    Top1 & Top2 --> Prune["Sub-Node Paragraph Pruning (> 1200 chars)"]
    Prune --> PagePin["Pin Exact Inner-Page: active_page from --- [PAGE X] ---"]
    PagePin --> PostFilter["Trailing Noise Filter (Drop nodes < 0.22 of Top Score)"]

    PostFilter --> MathCheck{"Contains Numerical Ratio or Calculation?"}
    MathCheck -->|"Yes"| ASTMath["Safe AST Math Evaluator (AST Whitelist, Zero eval)"]
    MathCheck -->|"No"| Synthesis["Grounded LLM Synthesis Stream"]
    ASTMath --> Synthesis

    Synthesis --> Output["Answer with Verifiable Citations (Section: 'Title', Page: X)"]

🚀 Key Features & Capabilities

1. 📄 Page Boundary Injection & Exact Inner-Page Pinning

Marker Format: Injects explicit --- [PAGE X] --- tokens during document ingestion for every physical page transition.
Sub-Node Paragraph Pruning: Long sections (> 1,200 characters) are parsed into individual paragraphs, scoring each paragraph against query keywords, phrase bi-grams, and numerical entities.
Inner Active Page Tracking: Tracks the exact preceding page marker for the winning paragraph, exposing node["active_page"] to LLMService.
Result: Raises Citation Page Accuracy from 38.33% to 83.13%, eliminating imprecise section-level citations in multi-page sections.

2. 🌲 Vectorless Hybrid Search (Tree + BM25 + Nucleus RRF)

100% Vectorless: Consumes zero vector storage, requires no vector database (Qdrant, Chroma, Pinecone), and eliminates cosine drift.
Sub-Millisecond Lexical Index: Powered by a zero-dependency Okapi BM25 implementation supporting Arabic and English tokenization, BM25 title boosting ($3\times$), and inverted index hashing in under 0.5ms.
Nucleus (top_p = 0.75) Cumulative Probability Cutoff: Normalizes consensus candidate scores into a probability distribution via Softmax ($T=0.15$), rejecting long-tail noise candidates ($P < 0.18$).
Result: Pushes Context Precision from 39.16% to 82.50% on IEEE benchmarks and 81.1% on the FinanceBench-style 10-K sample.

3. 🧠 Query Intent Routing & Collective Knowledge Dossiers

Sub-Millisecond Classifier: Classifies queries in $< 0.02\text{ms}$ into Local, Global / Overview, Multi-Hop, Comparison, or External Tool routes.
Dossier Injection: For global overview and cross-chapter synthesis queries, Atlas automatically injects the document's pre-computed Executive Dossier from MemoryService alongside root chapter summaries.
Result: Boosts Global Questions Retrieval Recall from 41.0% to 52.6%.

4. 🎓 Interactive Study Studio (NotebookLM-Style)

Interactive Mind Map (GET /studio/{doc_id}/mindmap): Automatically transforms the document's hierarchical outline tree and key concept findings into a visual node-edge graph with section titles and page numbers.
AI Quiz & Flashcard Generator (POST /studio/{doc_id}/quiz): Generates 5 high-yield Multiple Choice Questions (with options, correct answer, and explanation) and 5 Flashcards (front/back), each pinned to its exact (Section, Page) source citation.
Frontend Workspace Integration: Deep interactive viewer tab allowing users to click graph nodes to jump directly to document sections.

5. 🌐 Research & Study FastMCP Suite (5 Zero-Key Tools)

Mounted via official Model Context Protocol (FastMCP) at /mcp and exposed through REST endpoints with zero paid API key dependencies:

Deep Web Reader (read_webpage_markdown): Fetches full web pages using HTTPX and cleans HTML into readable Markdown via BeautifulSoup/Trafilatura with a strict 6-second timeout.
Academic & arXiv Tool (search_arxiv_papers): Queries the public arXiv Atom API (export.arxiv.org) for preprint titles, authors, abstracts, and direct PDF links.
YouTube Lecture Transcript (fetch_youtube_transcript): Extracts video IDs from any YouTube URL pattern, fetches transcripts, and segments them into 3-minute study chapters.
Safe AST Math Evaluator (evaluate_math_expression): Evaluates financial ratios, YoY growth rates, percentages, and math expressions using an Abstract Syntax Tree (AST) whitelist—blocking eval(), exec(), and malicious injection vectors.
Atlas Internal MCP Tools (atlas_*): Allows external agents (Cursor, Claude Desktop, Antigravity) to list indexed documents, query trees, and fetch executive knowledge dossiers directly.

6. 💾 Enterprise Persistence & Multi-Tier Memory

PostgreSQL / Supabase: Connected via threaded connection pooling (psycopg2.pool.ThreadedConnectionPool, 5 to 20 connections) on AWS eu-west-1.
Three-Tier Isolated Memory:
- Tier 1 (Short-Term): Sliding conversational window per chat session.
- Tier 2 (Long-Term): Rolling structured summaries preventing prompt bloat.
- Tier 3 (Workspace Knowledge): Pre-computed per-document dossiers and unified workspace synthesis.
Resilient Fallback: Gracefully degrades to local SQLite (memory.db) if cloud persistence is temporarily unreachable.

7. 🛡️ Resilient Dual LLM Inference Engine (HF + Groq)

Primary Provider: Hugging Face Serverless Inference API (meta-llama/Llama-3.1-8B-Instruct).
Zero-Downtime Fallback: Instantaneous automated failover to Groq Cloud API (qwen/qwen3.8-27b) upon token credit exhaustion (402 Payment Required), rate limits (429), or timeouts.

📊 Industry Benchmarks & Quantitative Evaluation (95 Questions)

Atlas was systematically validated across 95 comprehensive evaluation questions combining an 80-question IEEE Research Benchmark (data.pdf: Unifying LLMs and Knowledge Graphs: A Roadmap, IEEE TKDE 2024, 28 pages) with a 15-question FinanceBench-style 10-K sample (financebench_10k_sample.pdf, 6 pages).

Subsystem Health & Functional Audit (Table 1)

All 9 integrated subsystems, tools, and routers passed automated verification with strict latency bounds:

Subsystem / Tool	Audit Status	Latency	Operational Description
Ingestion & Page Boundary Injection	✅ PASS	1,344.9 ms	Ingested 6 pages, 6 sections, 100% page marker coverage
Vectorless Hybrid Retrieval (BM25 + RRF + Pruning)	✅ PASS	2.2 ms	BM25 search 0.47ms, fused top-4 nodes, pruning preserved page markers
MCP: Safe Math Evaluator (`evaluate_math_expression`)	✅ PASS	0.6 ms	AST-whitelisted ratios & exploit blocking verified
MCP: Academic & arXiv Tool (`search_arxiv_papers`)	✅ PASS	8,203.5 ms	Queried public arXiv Atom API (zero paid API keys)
MCP: Deep Web Reader (`read_webpage_markdown`)	✅ PASS	80.6 ms	Clean Markdown extraction with 6s safeguard timeout
MCP: YouTube Transcript (`fetch_youtube_transcript`)	✅ PASS	0.3 ms	URL extractor & 3-minute chapter segmentation
*MCP: Atlas Internal Tools (`atlas_`)**	✅ PASS	856.8 ms	List documents, grounded query, and executive dossier
Studio: Interactive Mindmap (`GET /studio/.../mindmap`)	✅ PASS	301.4 ms	Generated node-edge graph (14 nodes, 13 edges)
Studio: Quiz Generator (`POST /studio/.../quiz`)	✅ PASS	1,538.7 ms	Generated 5 MCQs + 5 Flashcards with exact `(Section, Page)` citations

IEEE TKDE Research Benchmark — 80 Questions (Table 2)

Comparison between baseline PageIndex RAG and the upgraded Atlas platform featuring Nucleus (top_p = 0.75) RRF and exact inner-page citation pinning:

Evaluation Metric	Baseline Score (Pre-Upgrades)	Upgraded Atlas (`top_p` Nucleus)	Delta (pp)	Benchmark Status
Context Precision	39.16%	82.50%	+43.34 pp	🎯 +80% Target Achieved
Citation Page Accuracy	38.33%	83.13%	+44.80 pp	🎯 +80% Target Achieved
Retrieval Recall	73.08%	77.98%	+4.90 pp	🚀 Major Uplift
Answer Correctness	87.73%	90.77%	+3.04 pp	✅ SOTA Quality
Faithfulness (Anti-Hallucination)	90.65%	92.01%	+1.36 pp	✅ Strictly Grounded
Global Questions Recall	41.00%	52.60%	+11.60 pp	🏆 Dossier Boost
Average Query Latency	1.80 s	1.45 s	-0.35 s	⚡ High Velocity
Average Cost per Query	$0.000919	$0.000512	-$0.000407	💰 Zero Embedding Cost

Category Performance Breakdown (IEEE Benchmark, N=80)

Summary Questions (N=10): 100.0% Context Precision, 100.0% Citation Accuracy, 83.7% Recall.
Multi-Hop Questions (N=20): 90.0% Context Precision, 85.0% Citation Accuracy, 87.5% Recall.
Numerical / Factual Questions (N=10): 90.0% Context Precision, 90.0% Citation Accuracy, 90.0% Recall.
Local Questions (N=20): 85.0% Context Precision, 85.0% Citation Accuracy, 85.0% Recall.
Global Overview Questions (N=20): 60.0% Context Precision, 67.5% Citation Accuracy, 52.6% Recall.

FinanceBench-style 10-K Sample — 15 Questions (Table 3)

Evaluated against financebench_10k_sample.pdf (Atlas Technologies, Inc. SEC Form 10-K Annual Report). This dataset is a small synthetic sample (N=15, 3 questions per category), so category-level numbers are indicative, not statistically robust.

Financial Category	N	Retrieval Recall	Context Precision	Answer Correctness	Faithfulness	Citation Page Acc	Avg Latency	Cost / Query
Overall (All 15 Questions)	15	84.4%	81.1%	94.7%	96.9%	81.1%	1.25 s	$0.000406
Table & Numerical Extraction	3	66.7%	66.7%	83.3%	88.7%	66.7%	—	—
Financial Ratio & Math Reasoning	3	100.0%	100.0%	100.0%	100.0%	100.0%	—	—
Multi-Hop Footnote Synthesis	3	83.3%	61.1%	95.0%	98.0%	61.1%	—	—
Global Executive Summary	3	72.2%	77.8%	95.0%	98.0%	77.8%	—	—
Out-of-Document / Live Web Fallback	3	100.0%	100.0%	100.0%	100.0%	100.0%	—	—

pie title FinanceBench-style 10-K Sample Question Distribution by Category
    "Table & Numerical Extraction" : 3
    "Financial Ratio & Math Reasoning" : 3
    "Multi-Hop Footnote Synthesis" : 3
    "Global Executive Summary" : 3
    "Out-of-Document / Live Web Fallback" : 3

📄 Full Benchmark Report: evaluation/FULL_BENCHMARK_AND_FINANCEBENCH_REPORT.md
📊 FinanceBench-style 10-K Sample Dataset: evaluation/financebench_sample.json
🧪 Master Evaluation Runner: evaluation/run_full_system_benchmark.py

🛠️ Tech Stack

Layer	Technologies & Libraries	Key Responsibilities
API Gateway	FastAPI (Python 3.11, async runtime)	High-performance asynchronous REST endpoints, SSE streams, CORS
Package Management	`uv` (Astral)	Sub-second deterministic dependency resolution & virtual environments
Vectorless Lexical Engine	Zero-Dependency Okapi BM25	In-memory tokenized index, Arabic/English normalization, title boost
RRF & Nucleus Selection	Weighted consensus RRF + Softmax	Dynamic `top_p = 0.75` cumulative cutoff, $T=0.15$, noise filtering
MCP Server Mount	`fastmcp` (4.1.0) + `mcp` (2.3.0)	Official Model Context Protocol server mounted via ASGI at `/mcp` (`>=4.1.0` and `>=2.3.0` in `pyproject.toml`, pinned in `uv.lock`)
Research Tools	`httpx`, `beautifulsoup4`, `trafilatura`	Clean HTML-to-Markdown extraction, public arXiv Atom XML parser
Media & Math Analysis	`youtube_transcript_api`, Python `ast`	Video transcript segmenter, exploit-safe AST math evaluator
Primary LLM Provider	`huggingface-hub` (`meta-llama/Llama-3.1-8B-Instruct`)	Outline tree navigation, sub-node pruning synthesis, citations
Fallback LLM Provider	`openai` (`qwen/qwen3.8-27b` via Groq Cloud)	Automated sub-second zero-downtime failover engine (~0.20s TTFT)
Document Ingestion	`PageIndexClient` + `PyMuPDF` (fitz)	Hierarchical outline extraction, `--- [PAGE X] ---` injection
Cloud Persistence	PostgreSQL (Supabase) + `psycopg2-binary`	Connection pooled storage for trees (JSONB), chats, and dossiers
Offline Fallback DB	SQLite (`memory.db`)	Local database fallback if cloud database is unreachable
Frontend UI	React 19 + Vite + Tailwind CSS 4	Interactive workspace, study studio, mind maps, real-time SSE
Deployment	Docker on Hugging Face Spaces + Vercel	Containerized backend server + global edge-deployed SPA client

🔌 API Endpoints Reference

1. Research & Chat

POST /ask — Stream grounded answers across single/multiple documents, or conversational mode with FastMCP fallback.
POST /chat/new — Create a new isolated chat session.
GET /chats — List all persisted chat sessions with unread status indicators.
GET /chat/{chat_id}/history — Retrieve full conversation turn history for a session.
POST /chat/{chat_id}/attach — Attach an indexed document to a chat session.
POST /chat/{chat_id}/attach-multiple — Attach multiple documents simultaneously.
GET /chat/{chat_id}/documents — List all documents attached to a session.
DELETE /chat/{chat_id}/document/{doc_id} — Detach a specific document from a session.
DELETE /chat/{chat_id} — Permanently delete a conversation session.
PATCH /chat/{chat_id}/title — Rename a conversation session title.

2. Study Studio (NotebookLM-Style)

GET /studio/{doc_id}/mindmap — Transform document outline tree and key concept findings into a visual node-edge JSON graph with section titles and page numbers.
POST /studio/{doc_id}/quiz — Generate 5 Multiple Choice Questions and 5 Flashcards with exact (Section, Page) citations.

3. FastMCP Research Tools

GET/POST /mcp — FastMCP server endpoint supporting Model Context Protocol via SSE/HTTP.
POST /tools/web-search — Direct REST endpoint for live DuckDuckGo web searches (query, max_results).
POST /tools/arxiv-search — Query public arXiv Atom API for academic preprints.
POST /tools/read-webpage — Fetch URL and extract clean Markdown with a 6-second timeout.
POST /tools/youtube-transcript — Extract transcripts and segment YouTube lectures into study chapters.
POST /tools/calculate-math — Execute AST-whitelisted financial and mathematical formulas safely.

4. Document Library & Tree Ingestion

POST /upload — Upload PDF to trigger asynchronous hierarchical tree indexing (HTTP 202).
GET /documents — List all ingested documents, page counts, and indexing statuses.
GET /documents/{document_id} — Get metadata, page count, and status for a document.
GET /documents/{document_id}/tree — Retrieve the full hierarchical outline tree.
DELETE /documents/{document_id} — Permanently delete a document and its cached indices.
POST /documents/{document_id}/retry — Retry indexing for a failed document.

5. Memory & Collective Knowledge

GET /memory/knowledge — Retrieve workspace master knowledge summary and all per-document dossiers.
POST /memory/knowledge/refresh — Synthesize and rebuild workspace global knowledge memory.
GET /memory/dossier/{doc_id} — Fetch structured knowledge dossier for an individual document.
GET /memory/chat/{chat_id} — Inspect short-term turns and long-term rolling summary.
POST /memory/chat/{chat_id}/summarize — Force-generate or update session long-term memory.

6. Health & Diagnostics

GET /health — Check API status, database connectivity, FastMCP status, and indexed document counts.

⚙️ Environment Variables

Configure these variables in your .env file or in Hugging Face Space Settings → Variables and secrets:

# Database Persistence (Supabase PostgreSQL on AWS eu-west-1)
DATABASE_URL="postgresql://postgres.USER:PASSWORD@aws-0-eu-west-1.pooler.supabase.com:5432/postgres"

# Hugging Face Inference Token (Primary Provider)
HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

# Groq Cloud API (Automatic Zero-Downtime Fallback)
GROQ_API_KEY="gsk_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
GROQ_CHAT_MODEL="qwen/qwen3.8-27b"
GROQ_RETRIEVAL_MODEL="qwen/qwen3.8-27b"

# PageIndex Ingestion API Keys (Rotated for High Throughput)
PAGEINDEX_API_KEY_1="your_pageindex_api_key_1"
PAGEINDEX_API_KEY_2="your_pageindex_api_key_2"
PAGEINDEX_API_KEY_3="your_pageindex_api_key_3"

# Optional Model Selection
HF_RETRIEVAL_MODEL="meta-llama/Llama-3.1-8B-Instruct"
HF_CHAT_MODEL="meta-llama/Llama-3.1-8B-Instruct"

🌐 Live Deployments

Frontend Application (Vercel): https://vectorless-front-8gpg.vercel.app
Backend API (Hugging Face Spaces): https://amer003100-vectorless-rag.hf.space
Interactive Swagger Docs: https://amer003100-vectorless-rag.hf.space/docs
FastMCP Endpoint: https://amer003100-vectorless-rag.hf.space/mcp