README.md

metadata

title: Multimodal RAG
emoji: 📄
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 6.17.3
python_version: '3.12'
startup_duration_timeout: 1h
app_file: app.py
pinned: false
license: apache-2.0
short_description: Ask questions over PDFs, Word files, images and charts
models:
  - Qwen/Qwen3-VL-4B-Instruct
  - BAAI/bge-small-en-v1.5

Multimodal RAG on ZeroGPU

Upload PDFs, Word files, images, screenshots or scanned pages and ask questions about them. The answer cites the text passages and images it used, and the retrieved images appear in a gallery.

Stage	Model / tool	Runs on
Parsing	PyMuPDF: text, embedded images, vector-drawn pages rendered, scanned pages rasterised. python-docx: text, tables, images	CPU
OCR (text inside images, charts, scans)	RapidOCR (PP-OCR in ONNX)	CPU
Text retrieval	BAAI/bge-small-en-v1.5 + BM25, fused with Reciprocal Rank Fusion	CPU
Image retrieval	OpenCLIP ViT-B/32 (laion400m_e32) text→image, OCR-linked images and images of vector-drawn pages linked to retrieved native text	CPU
Vector store	Qdrant in-memory, one index per browser session	CPU
Answer	Qwen/Qwen3-VL-4B-Instruct, which sees the numbered passages and the retrieved images	ZeroGPU (only while answering)

Each visitor receives a temporary session-isolated vector index with a 3-hour state TTL. Gradio may retain uploaded-file cache copies beyond this cleanup; avoid uploading confidential documents to a public Space.

Deploy

1. On huggingface.co, go to New Space, choose SDK: Gradio and Hardware: ZeroGPU. Eligible free personal accounts (verified email, account older than 30 days, good standing) can host up to two ZeroGPU Spaces; PRO is not required. Check current eligibility at https://huggingface.co/docs/hub/spaces-zerogpu.
2. Extract the ZIP and upload its contents to the Space repository root. app.py, README.md, requirements.txt, packages.txt, mrag/ and examples/ must sit at the root; do not upload only the ZIP or put everything inside an extra enclosing folder. Alternatively, from the extracted folder:

   pip install -U huggingface_hub
   hf auth login
   hf upload <user>/<space-name> . --repo-type=space
   

3. Wait for the Space to show Running. The first boot downloads Qwen3-VL-4B (about 9 GB), BGE-small and CLIP. The startup timeout is one hour to allow a cold download. A demo corpus is included (and can be regenerated), so visitors can click Load demo documents and ask straight away.
4. Ask the five example questions. Expected evidence: warranty 36 months (scanned PDF p.3); Asia-Pacific, USD 41.2 million (chart on PDF p.2); forklift 2500 kg (Word table); code rotation Monday at 09:00 (screenshot); office-cat photo (image retrieval). These are synthetic documents.
5. In a second browser session, verify that asking before indexing raises an error. Upload a small file there and verify it does not retrieve documents from the first session. Clear an index and confirm that answering requires indexing again.

No API key, Ollama service or external Qdrant server is required. Select ZeroGPU in the actual Space hardware settings; README metadata alone does not provision it.

Space variables (Settings → Variables), all optional

Variable	Default
MRAG_VLM_ID	Qwen/Qwen3-VL-4B-Instruct	e.g. Qwen/Qwen3-VL-8B-Instruct for better answers, Qwen/Qwen3-VL-2B-Instruct for speed
MRAG_TEXT_MODEL	BAAI/bge-small-en-v1.5	BAAI/bge-m3 for multilingual documents
MRAG_MAX_FILES / MRAG_MAX_FILE_MB / MRAG_MAX_PDF_PAGES	10 / 25 / 40	abuse limits for the public demo
MRAG_TOP_K_TEXT / MRAG_TOP_K_IMAGES	6 / 3	default context size (also adjustable in the UI)
MRAG_LLM_BACKEND	transformers	Keep this value on ZeroGPU. extractive is for local/CPU-only copies with no LLM.

The Gradio event chain runs retrieval on CPU, then calls the directly registered @spaces.GPU handler for streaming, only if retrieval succeeds. CPU callbacks share a one-task queue, GPU generation has concurrency one, and a session cannot index or clear its files during an answer. On either generation success or failure, a web-process callback unlocks the controls and clears the pending prompt and image paths.

GPU time per answer is requested dynamically: 20 s, plus 4 s per image, plus 1 s per 12 tokens of the "Max new tokens" setting, capped at 120 s. This is an unbenchmarked estimate, not a measured runtime guarantee. Start with 256 output tokens if visitors hit a quota or duration error, then tune after representative hosted measurements. Lower requests get better queue priority.

The file count is cumulative across uploads and demo loading. PDFs are limited to their first 40 pages by default; the rest are not indexed. Upload size is checked at both the Gradio upload endpoint and indexing handler. The UI answers each question independently; the displayed transcript is not passed as conversational memory to retrieval or the model.

Run locally

pip install -r requirements.txt
python app.py                                  # same Gradio app; uses CUDA/MPS if present, else CPU

The package also works without Gradio:

• CLI: python -m mrag ingest docs/, then python -m mrag ask "..." --backend ollama|openai|transformers|extractive
• Streamlit UI: streamlit run mrag/streamlit_app.py

All settings are in mrag/config.py and can be overridden with MRAG_<FIELD> environment variables.

Tests

pip install -r requirements-dev.txt
pytest -q

The bundled test suite covers:

• loaders, OCR and retrieval on real models (correct top-1 text and images, plus negative queries that must return no image)
• re-ingest and the CLI
• the Ollama and OpenAI-compatible clients, against local servers that speak their wire protocols
• the transformers VLM path, using a tiny Qwen3-VL built on disk (same processor, image-token expansion, vision tower and streaming)
• this Gradio app driven over its API with gradio_client: demo documents, streamed answers, uploads, and isolation between sessions

To run the VLM and app tests against a real checkpoint, set MRAG_VLM_ID=Qwen/Qwen3-VL-2B-Instruct.

Changes from the original Claude ZIP

See REVIEW_AND_TEST_REPORT.md for the latest corrections, measured validation results and remaining hosted checks. Full hosted inference still needs to be tested on an actual ZeroGPU Space.