--- title: Atlas Document Intelligence API emoji: 🌿 colorFrom: green colorTo: blue sdk: docker app_port: 7860 pinned: false --- # 🌿 Atlas — Enterprise Vectorless RAG & Document Intelligence Platform > **Understand Documents. Discover Knowledge. Reason with Exact Page Verification.**
> Production-grade Vectorless RAG platform powered by hierarchical outline reasoning (`PageIndex` + `PyMuPDF`), pure-Python lexical BM25 with **Nucleus (`top_p`) Cumulative Probability Cutoff**, Sub-Node Paragraph Pruning with **Exact Inner-Page Citation Pinning**, an **Interactive Study Studio** (Mind Maps + Quizzes), a **Research & Study FastMCP Suite** (5 zero-paid-key tools), and resilient dual LLM inference (Hugging Face + Groq zero-downtime failover) on PostgreSQL persistence. --- ## 📑 جدول المحتويات / Table of Contents 1. [Overview & Core Philosophy](#-overview--core-philosophy) 2. [Complete System Architecture](#-complete-system-architecture) 3. [Vectorless Hybrid Retrieval & Nucleus (`top_p`) Cutoff](#-vectorless-hybrid-retrieval--nucleus-top_p-cutoff) 4. [Key Features & Capabilities](#-key-features--capabilities) - [1. Page Boundary Injection & Exact Inner-Page Pinning](#1-page-boundary-injection--exact-inner-page-pinning) - [2. Vectorless Hybrid Search (Tree + BM25 + Nucleus RRF)](#2-vectorless-hybrid-search-tree--bm25--nucleus-rrf) - [3. Query Intent Routing & Collective Knowledge Dossiers](#3-query-intent-routing--collective-knowledge-dossiers) - [4. Interactive Study Studio (Mind Maps + Quizzes)](#4-interactive-study-studio-notebooklm-style) - [5. Research & Study FastMCP Suite (5 Zero-Key Tools)](#5-research--study-fastmcp-suite-5-zero-key-tools) - [6. Enterprise Persistence & Multi-Tier Memory](#6-enterprise-persistence--multi-tier-memory) - [7. Resilient Dual LLM Inference Engine](#7-resilient-dual-llm-inference-engine-hf--groq) 5. [Industry Benchmarks & Quantitative Evaluation (95 Questions)](#-industry-benchmarks--quantitative-evaluation-95-questions) - [Subsystem Health & Functional Audit (Table 1)](#subsystem-health--functional-audit-table-1) - [IEEE TKDE Research Benchmark — 80 Questions (Table 2)](#ieee-tkde-research-benchmark--80-questions-table-2) - [FinanceBench-style 10-K Sample — 15 Questions (Table 3)](#financebench-style-10-k-sample--15-questions-table-3) 6. [Tech Stack](#-tech-stack) 7. [API Endpoints Reference](#-api-endpoints-reference) 8. [Environment Variables](#-environment-variables) 9. [Live Deployments](#-live-deployments) --- ## 🌟 Overview & Core Philosophy **Atlas** redefines Document Intelligence and Retrieval-Augmented Generation (RAG) by completely eliminating vector embeddings, arbitrary chunking, and similarity drift. Instead of blindly slicing documents into 500-token chunks and computing cosine distance over a vector database, Atlas models documents as **Hierarchical Semantic Trees** (`PageIndex` + `PyMuPDF`). Every section, chapter, table, and footnote preserves its parent-child lineage and exact physical page coordinates. When a user queries the system: 1. **Hybrid Retrieval**: Runs lexical BM25 search and weighted RRF fusion in ~2ms, while the LLM tree-search step adds model latency, achieving an end-to-end average query latency of 1.45s. 2. **Nucleus Consensus (`top_p = 0.75`)**: Combines rankings via Weighted Reciprocal Rank Fusion (RRF: `0.6` Tree, `0.4` BM25) and applies temperature-scaled Softmax to dynamically isolate the minimal sufficient set of nodes (single-hop queries truncate to 1 node; multi-hop queries preserve 2 nodes). 3. **Sub-Node Paragraph Pruning & Exact Inner-Page Pinning**: Filters long sections down to high-density paragraphs and pins citations to the exact `--- [PAGE X] ---` where evidence resides. 4. **Fallback & Extended Research**: When out-of-document facts, arXiv papers, mathematical ratios, or YouTube lectures are needed, Atlas activates its built-in **Research & Study FastMCP Suite** with zero external API fees. --- ## 🏛️ Complete System Architecture ```mermaid flowchart TB subgraph Client_Layer["🖥️ Frontend Client (React 19 + Vite / Vercel)"] direction TB UI_Sidebar["Sidebar (Sessions, Multi-Chat & Unread Indicators)"] UI_Explorer["Document Structure Explorer (Hierarchical Tree View)"] UI_Studio["Study Studio (Interactive Mind Map & Flashcard Quiz)"] UI_Chat["Concurrent Async Chat (Multi-Session Streaming)"] UI_Sources["Interactive Source Badges (📄 PDF Pages & 🌐 Web Badges)"] UI_Memory["Knowledge Memory & Workspace Dossier View"] UI_SSE["SSE Stream Parser & Background Dispatcher"] UI_Storage[("Browser LocalStorage Cache")] UI_Sidebar <--> UI_Chat UI_Chat <--> UI_Sources UI_Chat <--> UI_SSE UI_Chat <--> UI_Storage UI_Sidebar <--> UI_Studio end subgraph API_Layer["⚡ Backend Gateway (FastAPI / Hugging Face Spaces)"] direction TB Router_Chat["/ask & /chat Routers (Streaming SSE, Multi-Doc)"] Router_Studio["/studio Router (/mindmap, /quiz MCQs & Flashcards)"] Router_Tools["/tools Router (Web, arXiv, YouTube, Math)"] Router_Docs["/documents Router (Upload, Ingestion, Trees)"] Router_Memory["/memory Router (Dossiers, Workspace Synthesis)"] Router_MCP["FastMCP Server Mount (/mcp via SSE/HTTP)"] end subgraph FastMCP_Suite["🌐 Research & Study FastMCP Suite (Zero Paid Keys)"] direction LR Tool_Web["Deep Web Reader (httpx + BS4/Trafilatura, 6s timeout)"] Tool_ArXiv["Academic arXiv Tool (export.arxiv.org Atom API)"] Tool_YT["YouTube Transcript (Segmenter & Chapter Analyzer)"] Tool_Math["Safe AST Math Evaluator (AST Whitelist, Zero eval)"] Tool_Atlas["Atlas Internal Tools (list_docs, query, dossiers)"] end subgraph Core_Engine["🧠 Vectorless RAG & Reasoning Pipelines"] direction TB Intent_Classifier["Fast Intent Classifier (< 0.02ms Deterministic)"] Doc_Service["DocumentService (PageIndex API + PyMuPDF Page Markers)"] BM25_Engine["Zero-Dependency Okapi BM25 (Arabic/English Stemmer)"] RAG_Pipeline["VectorlessRAG (Weighted RRF + Nucleus top_p Softmax)"] Pruner["Sub-Node Paragraph Pruning & Exact Inner-Page Pinning"] LLM_Service["LLMService (Tree Search, Grounded Synthesis, Dual Failover)"] Memory_Service["MemoryService (Short-Term, Long-Term, Workspace Dossiers)"] end subgraph AI_Inference["🤖 Resilient Dual-Provider LLM Engine"] direction LR HF_LLM["Hugging Face Serverless (Primary: meta-llama/Llama-3.1-8B-Instruct)"] Groq_LLM["Groq Cloud API (Zero-Downtime Fallback: qwen/qwen3.8-27b)"] HF_LLM -.->|Auto-Failover on 402/Limit/Error| Groq_LLM end subgraph Persistence_Layer["💾 Enterprise Persistence & Database"] direction TB Mem_Store["MemoryStorage (In-Memory Hot Cache)"] subgraph DB_System["PostgreSQL / Supabase (AWS eu-west-1)"] PG_Pool["Threaded Connection Pool (psycopg2)"] T_Meta[("doc_meta (Metadata & Statuses)")] T_Indices[("doc_indices (JSONB Trees & Injected Markers)")] T_Chats[("chats & messages (Isolated Sessions)")] T_Docs[("chat_documents (Multi-Doc Associations)")] T_Memory[("knowledge_memory (Per-Doc Dossiers)")] T_Global[("global_knowledge_summary (Workspace)")] end SQLite_Fallback[("SQLite Local Fallback (memory.db)")] end %% Wiring Client_Layer <==>|REST API & SSE Streams| API_Layer API_Layer <--> Router_MCP Router_MCP <--> FastMCP_Suite Router_Studio --> LLM_Service Router_Chat --> Intent_Classifier Intent_Classifier --> RAG_Pipeline RAG_Pipeline --> BM25_Engine & Doc_Service RAG_Pipeline --> Pruner Pruner --> LLM_Service LLM_Service <==> AI_Inference Router_Memory --> Memory_Service Mem_Store <--> PG_Pool PG_Pool --> T_Meta & T_Indices & T_Chats & T_Docs & T_Memory & T_Global PG_Pool -.->|Offline Failover| SQLite_Fallback ``` --- ## 🧭 Vectorless Hybrid Retrieval & Nucleus (`top_p`) Cutoff Instead of relying on arbitrary static `top_k=3` cutoffs that pollute LLM context on single-fact queries, Atlas dynamically sizes the retrieved node set via **Temperature-Scaled Softmax Nucleus Retrieval**: ```mermaid flowchart TD Q["User Query"] --> Clf{"Fast Intent Classifier (< 0.02ms)"} Clf -->|"Academic Paper"| MCP_Arxiv["FastMCP arXiv Tool (export.arxiv.org)"] Clf -->|"Live Market / Web"| MCP_Web["FastMCP Web Reader (Clean Markdown)"] Clf -->|"Global / Overview"| Dossier["Pre-computed Executive Dossier + Root Chapters"] Clf -->|"Specific / Numerical"| Hybrid["Parallel Hybrid Search (BM25 + Fusion ~2ms)"] Hybrid --> Tree["Hierarchical LLM Tree Traversal (Weight 0.6)"] Hybrid --> BM25["Pure-Python Okapi BM25 (Weight 0.4)"] Tree & BM25 --> Consensus["Consensus Score: S(d) = sum w_L / (60 + rank_L) * Density"] Consensus --> Softmax["Temperature-Scaled Softmax (T = 0.15): P(node_i)"] Softmax --> Nucleus{"Accumulate P(node_i) until sum >= top_p (0.75)"} Nucleus -->|Single-Node Local Query| Top1["Select Minimal Sufficient Top-1 Node"] Nucleus -->|Multi-Hop Complex Query| Top2["Preserve High-Relevance Top-2 Nodes (min_k=2)"] Top1 & Top2 --> Prune["Sub-Node Paragraph Pruning (> 1200 chars)"] Prune --> PagePin["Pin Exact Inner-Page: active_page from --- [PAGE X] ---"] PagePin --> PostFilter["Trailing Noise Filter (Drop nodes < 0.22 of Top Score)"] PostFilter --> MathCheck{"Contains Numerical Ratio or Calculation?"} MathCheck -->|"Yes"| ASTMath["Safe AST Math Evaluator (AST Whitelist, Zero eval)"] MathCheck -->|"No"| Synthesis["Grounded LLM Synthesis Stream"] ASTMath --> Synthesis Synthesis --> Output["Answer with Verifiable Citations (Section: 'Title', Page: X)"] ``` --- ## 🚀 Key Features & Capabilities ### 1. 📄 Page Boundary Injection & Exact Inner-Page Pinning - **Marker Format**: Injects explicit `--- [PAGE X] ---` tokens during document ingestion for every physical page transition. - **Sub-Node Paragraph Pruning**: Long sections (> 1,200 characters) are parsed into individual paragraphs, scoring each paragraph against query keywords, phrase bi-grams, and numerical entities. - **Inner Active Page Tracking**: Tracks the exact preceding page marker for the winning paragraph, exposing `node["active_page"]` to `LLMService`. - **Result**: Raises Citation Page Accuracy from **38.33% to 83.13%**, eliminating imprecise section-level citations in multi-page sections. ### 2. 🌲 Vectorless Hybrid Search (Tree + BM25 + Nucleus RRF) - **100% Vectorless**: Consumes zero vector storage, requires no vector database (Qdrant, Chroma, Pinecone), and eliminates cosine drift. - **Sub-Millisecond Lexical Index**: Powered by a zero-dependency Okapi BM25 implementation supporting Arabic and English tokenization, BM25 title boosting ($3\times$), and inverted index hashing in under **0.5ms**. - **Nucleus (`top_p = 0.75`) Cumulative Probability Cutoff**: Normalizes consensus candidate scores into a probability distribution via Softmax ($T=0.15$), rejecting long-tail noise candidates ($P < 0.18$). - **Result**: Pushes Context Precision from **39.16% to 82.50%** on IEEE benchmarks and **81.1%** on the FinanceBench-style 10-K sample. ### 3. 🧠 Query Intent Routing & Collective Knowledge Dossiers - **Sub-Millisecond Classifier**: Classifies queries in $< 0.02\text{ms}$ into `Local`, `Global / Overview`, `Multi-Hop`, `Comparison`, or `External Tool` routes. - **Dossier Injection**: For global overview and cross-chapter synthesis queries, Atlas automatically injects the document's pre-computed Executive Dossier from `MemoryService` alongside root chapter summaries. - **Result**: Boosts Global Questions Retrieval Recall from **41.0% to 52.6%**. ### 4. 🎓 Interactive Study Studio (NotebookLM-Style) - **Interactive Mind Map (`GET /studio/{doc_id}/mindmap`)**: Automatically transforms the document's hierarchical outline tree and key concept findings into a visual node-edge graph with section titles and page numbers. - **AI Quiz & Flashcard Generator (`POST /studio/{doc_id}/quiz`)**: Generates 5 high-yield Multiple Choice Questions (with options, correct answer, and explanation) and 5 Flashcards (front/back), each pinned to its exact `(Section, Page)` source citation. - **Frontend Workspace Integration**: Deep interactive viewer tab allowing users to click graph nodes to jump directly to document sections. ### 5. 🌐 Research & Study FastMCP Suite (5 Zero-Key Tools) Mounted via official Model Context Protocol (FastMCP) at `/mcp` and exposed through REST endpoints with zero paid API key dependencies: 1. **Deep Web Reader (`read_webpage_markdown`)**: Fetches full web pages using HTTPX and cleans HTML into readable Markdown via BeautifulSoup/Trafilatura with a strict 6-second timeout. 2. **Academic & arXiv Tool (`search_arxiv_papers`)**: Queries the public arXiv Atom API (`export.arxiv.org`) for preprint titles, authors, abstracts, and direct PDF links. 3. **YouTube Lecture Transcript (`fetch_youtube_transcript`)**: Extracts video IDs from any YouTube URL pattern, fetches transcripts, and segments them into 3-minute study chapters. 4. **Safe AST Math Evaluator (`evaluate_math_expression`)**: Evaluates financial ratios, YoY growth rates, percentages, and math expressions using an Abstract Syntax Tree (AST) whitelist—blocking `eval()`, `exec()`, and malicious injection vectors. 5. **Atlas Internal MCP Tools (`atlas_*`)**: Allows external agents (Cursor, Claude Desktop, Antigravity) to list indexed documents, query trees, and fetch executive knowledge dossiers directly. ### 6. 💾 Enterprise Persistence & Multi-Tier Memory - **PostgreSQL / Supabase**: Connected via threaded connection pooling (`psycopg2.pool.ThreadedConnectionPool`, 5 to 20 connections) on AWS `eu-west-1`. - **Three-Tier Isolated Memory**: - *Tier 1 (Short-Term)*: Sliding conversational window per chat session. - *Tier 2 (Long-Term)*: Rolling structured summaries preventing prompt bloat. - *Tier 3 (Workspace Knowledge)*: Pre-computed per-document dossiers and unified workspace synthesis. - **Resilient Fallback**: Gracefully degrades to local SQLite (`memory.db`) if cloud persistence is temporarily unreachable. ### 7. 🛡️ Resilient Dual LLM Inference Engine (HF + Groq) - **Primary Provider**: Hugging Face Serverless Inference API (`meta-llama/Llama-3.1-8B-Instruct`). - **Zero-Downtime Fallback**: Instantaneous automated failover to Groq Cloud API (`qwen/qwen3.8-27b`) upon token credit exhaustion (`402 Payment Required`), rate limits (`429`), or timeouts. --- ## 📊 Industry Benchmarks & Quantitative Evaluation (95 Questions) Atlas was systematically validated across **95 comprehensive evaluation questions** combining an **80-question IEEE Research Benchmark** (`data.pdf`: *Unifying LLMs and Knowledge Graphs: A Roadmap*, IEEE TKDE 2024, 28 pages) with a **15-question FinanceBench-style 10-K sample** (`financebench_10k_sample.pdf`, 6 pages). ### Subsystem Health & Functional Audit (Table 1) All 9 integrated subsystems, tools, and routers passed automated verification with strict latency bounds: | Subsystem / Tool | Audit Status | Latency | Operational Description | | :--- | :---: | :---: | :--- | | **Ingestion & Page Boundary Injection** | ✅ PASS | 1,344.9 ms | Ingested 6 pages, 6 sections, 100% page marker coverage | | **Vectorless Hybrid Retrieval (BM25 + RRF + Pruning)** | ✅ PASS | 2.2 ms | BM25 search 0.47ms, fused top-4 nodes, pruning preserved page markers | | **MCP: Safe Math Evaluator (`evaluate_math_expression`)** | ✅ PASS | 0.6 ms | AST-whitelisted ratios & exploit blocking verified | | **MCP: Academic & arXiv Tool (`search_arxiv_papers`)** | ✅ PASS | 8,203.5 ms | Queried public arXiv Atom API (zero paid API keys) | | **MCP: Deep Web Reader (`read_webpage_markdown`)** | ✅ PASS | 80.6 ms | Clean Markdown extraction with 6s safeguard timeout | | **MCP: YouTube Transcript (`fetch_youtube_transcript`)** | ✅ PASS | 0.3 ms | URL extractor & 3-minute chapter segmentation | | **MCP: Atlas Internal Tools (`atlas_*`)** | ✅ PASS | 856.8 ms | List documents, grounded query, and executive dossier | | **Studio: Interactive Mindmap (`GET /studio/.../mindmap`)** | ✅ PASS | 301.4 ms | Generated node-edge graph (14 nodes, 13 edges) | | **Studio: Quiz Generator (`POST /studio/.../quiz`)** | ✅ PASS | 1,538.7 ms | Generated 5 MCQs + 5 Flashcards with exact `(Section, Page)` citations | --- ### IEEE TKDE Research Benchmark — 80 Questions (Table 2) Comparison between baseline PageIndex RAG and the upgraded Atlas platform featuring Nucleus (`top_p = 0.75`) RRF and exact inner-page citation pinning: | Evaluation Metric | Baseline Score (Pre-Upgrades) | Upgraded Atlas (`top_p` Nucleus) | Delta (pp) | Benchmark Status | | :--- | :---: | :---: | :---: | :---: | | **Context Precision** | 39.16% | **82.50%** | **+43.34 pp** | 🎯 **+80% Target Achieved** | | **Citation Page Accuracy** | 38.33% | **83.13%** | **+44.80 pp** | 🎯 **+80% Target Achieved** | | **Retrieval Recall** | 73.08% | **77.98%** | **+4.90 pp** | 🚀 Major Uplift | | **Answer Correctness** | 87.73% | **90.77%** | **+3.04 pp** | ✅ SOTA Quality | | **Faithfulness (Anti-Hallucination)** | 90.65% | **92.01%** | **+1.36 pp** | ✅ Strictly Grounded | | **Global Questions Recall** | 41.00% | **52.60%** | **+11.60 pp** | 🏆 Dossier Boost | | **Average Query Latency** | 1.80 s | **1.45 s** | **-0.35 s** | ⚡ High Velocity | | **Average Cost per Query** | $0.000919 | **$0.000512** | **-$0.000407** | 💰 Zero Embedding Cost | #### Category Performance Breakdown (IEEE Benchmark, N=80) - **Summary Questions (N=10)**: **100.0% Context Precision**, **100.0% Citation Accuracy**, **83.7% Recall**. - **Multi-Hop Questions (N=20)**: **90.0% Context Precision**, **85.0% Citation Accuracy**, **87.5% Recall**. - **Numerical / Factual Questions (N=10)**: **90.0% Context Precision**, **90.0% Citation Accuracy**, **90.0% Recall**. - **Local Questions (N=20)**: **85.0% Context Precision**, **85.0% Citation Accuracy**, **85.0% Recall**. - **Global Overview Questions (N=20)**: **60.0% Context Precision**, **67.5% Citation Accuracy**, **52.6% Recall**. --- ### FinanceBench-style 10-K Sample — 15 Questions (Table 3) Evaluated against `financebench_10k_sample.pdf` (*Atlas Technologies, Inc. SEC Form 10-K Annual Report*). This dataset is a small synthetic sample (N=15, 3 questions per category), so category-level numbers are indicative, not statistically robust. | Financial Category | N | Retrieval Recall | Context Precision | Answer Correctness | Faithfulness | Citation Page Acc | Avg Latency | Cost / Query | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | **Overall (All 15 Questions)** | 15 | **84.4%** | **81.1%** | **94.7%** | **96.9%** | **81.1%** | **1.25 s** | **$0.000406** | | **Table & Numerical Extraction** | 3 | 66.7% | 66.7% | 83.3% | 88.7% | 66.7% | — | — | | **Financial Ratio & Math Reasoning** | 3 | **100.0%** | **100.0%** | **100.0%** | **100.0%** | **100.0%** | — | — | | **Multi-Hop Footnote Synthesis** | 3 | 83.3% | 61.1% | 95.0% | 98.0% | 61.1% | — | — | | **Global Executive Summary** | 3 | 72.2% | 77.8% | 95.0% | 98.0% | 77.8% | — | — | | **Out-of-Document / Live Web Fallback** | 3 | **100.0%** | **100.0%** | **100.0%** | **100.0%** | **100.0%** | — | — | ```mermaid pie title FinanceBench-style 10-K Sample Question Distribution by Category "Table & Numerical Extraction" : 3 "Financial Ratio & Math Reasoning" : 3 "Multi-Hop Footnote Synthesis" : 3 "Global Executive Summary" : 3 "Out-of-Document / Live Web Fallback" : 3 ``` - 📄 **Full Benchmark Report:** [evaluation/FULL_BENCHMARK_AND_FINANCEBENCH_REPORT.md](evaluation/FULL_BENCHMARK_AND_FINANCEBENCH_REPORT.md) - 📊 **FinanceBench-style 10-K Sample Dataset:** [evaluation/financebench_sample.json](evaluation/financebench_sample.json) - 🧪 **Master Evaluation Runner:** [evaluation/run_full_system_benchmark.py](evaluation/run_full_system_benchmark.py) --- ## 🛠️ Tech Stack | Layer | Technologies & Libraries | Key Responsibilities | |---|---|---| | **API Gateway** | **FastAPI** (Python 3.11, async runtime) | High-performance asynchronous REST endpoints, SSE streams, CORS | | **Package Management** | **`uv`** (Astral) | Sub-second deterministic dependency resolution & virtual environments | | **Vectorless Lexical Engine** | **Zero-Dependency Okapi BM25** | In-memory tokenized index, Arabic/English normalization, title boost | | **RRF & Nucleus Selection** | **Weighted consensus RRF + Softmax** | Dynamic `top_p = 0.75` cumulative cutoff, $T=0.15$, noise filtering | | **MCP Server Mount** | **`fastmcp` (4.1.0)** + **`mcp` (2.3.0)** | Official Model Context Protocol server mounted via ASGI at `/mcp` (`>=4.1.0` and `>=2.3.0` in `pyproject.toml`, pinned in `uv.lock`) | | **Research Tools** | **`httpx`**, **`beautifulsoup4`**, **`trafilatura`** | Clean HTML-to-Markdown extraction, public arXiv Atom XML parser | | **Media & Math Analysis** | **`youtube_transcript_api`**, **Python `ast`** | Video transcript segmenter, exploit-safe AST math evaluator | | **Primary LLM Provider** | **`huggingface-hub`** (`meta-llama/Llama-3.1-8B-Instruct`) | Outline tree navigation, sub-node pruning synthesis, citations | | **Fallback LLM Provider** | **`openai`** (`qwen/qwen3.8-27b` via Groq Cloud) | Automated sub-second zero-downtime failover engine (~0.20s TTFT) | | **Document Ingestion** | **`PageIndexClient`** + **`PyMuPDF` (fitz)** | Hierarchical outline extraction, `--- [PAGE X] ---` injection | | **Cloud Persistence** | **PostgreSQL (Supabase)** + `psycopg2-binary` | Connection pooled storage for trees (JSONB), chats, and dossiers | | **Offline Fallback DB** | **SQLite** (`memory.db`) | Local database fallback if cloud database is unreachable | | **Frontend UI** | **React 19** + **Vite** + **Tailwind CSS 4** | Interactive workspace, study studio, mind maps, real-time SSE | | **Deployment** | **Docker** on Hugging Face Spaces + **Vercel** | Containerized backend server + global edge-deployed SPA client | --- ## 🔌 API Endpoints Reference ### 1. Research & Chat - `POST /ask` — Stream grounded answers across single/multiple documents, or conversational mode with FastMCP fallback. - `POST /chat/new` — Create a new isolated chat session. - `GET /chats` — List all persisted chat sessions with unread status indicators. - `GET /chat/{chat_id}/history` — Retrieve full conversation turn history for a session. - `POST /chat/{chat_id}/attach` — Attach an indexed document to a chat session. - `POST /chat/{chat_id}/attach-multiple` — Attach multiple documents simultaneously. - `GET /chat/{chat_id}/documents` — List all documents attached to a session. - `DELETE /chat/{chat_id}/document/{doc_id}` — Detach a specific document from a session. - `DELETE /chat/{chat_id}` — Permanently delete a conversation session. - `PATCH /chat/{chat_id}/title` — Rename a conversation session title. ### 2. Study Studio (NotebookLM-Style) - `GET /studio/{doc_id}/mindmap` — Transform document outline tree and key concept findings into a visual node-edge JSON graph with section titles and page numbers. - `POST /studio/{doc_id}/quiz` — Generate 5 Multiple Choice Questions and 5 Flashcards with exact `(Section, Page)` citations. ### 3. FastMCP Research Tools - `GET/POST /mcp` — FastMCP server endpoint supporting Model Context Protocol via SSE/HTTP. - `POST /tools/web-search` — Direct REST endpoint for live DuckDuckGo web searches (`query`, `max_results`). - `POST /tools/arxiv-search` — Query public arXiv Atom API for academic preprints. - `POST /tools/read-webpage` — Fetch URL and extract clean Markdown with a 6-second timeout. - `POST /tools/youtube-transcript` — Extract transcripts and segment YouTube lectures into study chapters. - `POST /tools/calculate-math` — Execute AST-whitelisted financial and mathematical formulas safely. ### 4. Document Library & Tree Ingestion - `POST /upload` — Upload PDF to trigger asynchronous hierarchical tree indexing (HTTP 202). - `GET /documents` — List all ingested documents, page counts, and indexing statuses. - `GET /documents/{document_id}` — Get metadata, page count, and status for a document. - `GET /documents/{document_id}/tree` — Retrieve the full hierarchical outline tree. - `DELETE /documents/{document_id}` — Permanently delete a document and its cached indices. - `POST /documents/{document_id}/retry` — Retry indexing for a failed document. ### 5. Memory & Collective Knowledge - `GET /memory/knowledge` — Retrieve workspace master knowledge summary and all per-document dossiers. - `POST /memory/knowledge/refresh` — Synthesize and rebuild workspace global knowledge memory. - `GET /memory/dossier/{doc_id}` — Fetch structured knowledge dossier for an individual document. - `GET /memory/chat/{chat_id}` — Inspect short-term turns and long-term rolling summary. - `POST /memory/chat/{chat_id}/summarize` — Force-generate or update session long-term memory. ### 6. Health & Diagnostics - `GET /health` — Check API status, database connectivity, FastMCP status, and indexed document counts. --- ## ⚙️ Environment Variables Configure these variables in your `.env` file or in **Hugging Face Space Settings → Variables and secrets**: ```env # Database Persistence (Supabase PostgreSQL on AWS eu-west-1) DATABASE_URL="postgresql://postgres.USER:PASSWORD@aws-0-eu-west-1.pooler.supabase.com:5432/postgres" # Hugging Face Inference Token (Primary Provider) HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" # Groq Cloud API (Automatic Zero-Downtime Fallback) GROQ_API_KEY="gsk_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" GROQ_CHAT_MODEL="qwen/qwen3.8-27b" GROQ_RETRIEVAL_MODEL="qwen/qwen3.8-27b" # PageIndex Ingestion API Keys (Rotated for High Throughput) PAGEINDEX_API_KEY_1="your_pageindex_api_key_1" PAGEINDEX_API_KEY_2="your_pageindex_api_key_2" PAGEINDEX_API_KEY_3="your_pageindex_api_key_3" # Optional Model Selection HF_RETRIEVAL_MODEL="meta-llama/Llama-3.1-8B-Instruct" HF_CHAT_MODEL="meta-llama/Llama-3.1-8B-Instruct" ``` --- ## 🌐 Live Deployments - **Frontend Application (Vercel):** [https://vectorless-front-8gpg.vercel.app](https://vectorless-front-8gpg.vercel.app) - **Backend API (Hugging Face Spaces):** [https://amer003100-vectorless-rag.hf.space](https://amer003100-vectorless-rag.hf.space) - **Interactive Swagger Docs:** [https://amer003100-vectorless-rag.hf.space/docs](https://amer003100-vectorless-rag.hf.space/docs) - **FastMCP Endpoint:** `https://amer003100-vectorless-rag.hf.space/mcp`