---
title: Multimodal RAG
emoji: 📄
colorFrom: indigo
colorTo: blue
sdk: gradio
sdk_version: 6.17.3
python_version: "3.12"
startup_duration_timeout: 1h
app_file: app.py
pinned: false
license: apache-2.0
short_description: Ask questions over PDFs, Word files, images and charts
models:
  - Qwen/Qwen3-VL-4B-Instruct
  - BAAI/bge-small-en-v1.5
---

# Multimodal RAG on ZeroGPU

Upload PDFs, Word files, images, screenshots or scanned pages and ask questions about them. The answer cites the text passages and images it used, and the retrieved images appear in a gallery.

| Stage | Model / tool | Runs on |
|---|---|---|
| Parsing | PyMuPDF: text, embedded images, vector-drawn pages rendered, scanned pages rasterised. python-docx: text, tables, images | CPU |
| OCR (text inside images, charts, scans) | RapidOCR (PP-OCR in ONNX) | CPU |
| Text retrieval | `BAAI/bge-small-en-v1.5` + BM25, fused with Reciprocal Rank Fusion | CPU |
| Image retrieval | OpenCLIP ViT-B/32 (`laion400m_e32`) text→image, OCR-linked images and images of vector-drawn pages linked to retrieved native text | CPU |
| Vector store | Qdrant in-memory, one index per browser session | CPU |
| Answer | `Qwen/Qwen3-VL-4B-Instruct`, which sees the numbered passages **and the retrieved images** | ZeroGPU (only while answering) |

Each visitor receives a temporary session-isolated vector index with a 3-hour state TTL. Gradio may retain uploaded-file cache copies beyond this cleanup; avoid uploading confidential documents to a public Space.

## Deploy

1. On huggingface.co, go to **New Space**, choose **SDK: Gradio** and **Hardware: ZeroGPU**. Eligible free personal accounts (verified email, account older than 30 days, good standing) can host up to two ZeroGPU Spaces; PRO is not required. Check current eligibility at https://huggingface.co/docs/hub/spaces-zerogpu.
2. Extract the ZIP and upload its **contents** to the Space repository root. `app.py`, `README.md`, `requirements.txt`, `packages.txt`, `mrag/` and `examples/` must sit at the root; do not upload only the ZIP or put everything inside an extra enclosing folder. Alternatively, from the extracted folder:
   ```bash
   pip install -U huggingface_hub
   hf auth login
   hf upload <user>/<space-name> . --repo-type=space
   ```
3. Wait for the Space to show **Running**. The first boot downloads Qwen3-VL-4B (about 9 GB), BGE-small and CLIP. The startup timeout is one hour to allow a cold download. A demo corpus is included (and can be regenerated), so visitors can click **Load demo documents** and ask straight away.
4. Ask the five example questions. Expected evidence: warranty **36 months** (scanned PDF p.3); **Asia-Pacific, USD 41.2 million** (chart on PDF p.2); forklift **2500 kg** (Word table); code rotation **Monday at 09:00** (screenshot); office-cat photo (image retrieval). These are synthetic documents.
5. In a second browser session, verify that asking before indexing raises an error. Upload a small file there and verify it does not retrieve documents from the first session. Clear an index and confirm that answering requires indexing again.

No API key, Ollama service or external Qdrant server is required. Select ZeroGPU in the actual Space hardware settings; README metadata alone does not provision it.

### Space variables (Settings → Variables), all optional

| Variable | Default | |
|---|---|---|
| `MRAG_VLM_ID` | `Qwen/Qwen3-VL-4B-Instruct` | e.g. `Qwen/Qwen3-VL-8B-Instruct` for better answers, `Qwen/Qwen3-VL-2B-Instruct` for speed |
| `MRAG_TEXT_MODEL` | `BAAI/bge-small-en-v1.5` | `BAAI/bge-m3` for multilingual documents |
| `MRAG_MAX_FILES` / `MRAG_MAX_FILE_MB` / `MRAG_MAX_PDF_PAGES` | 10 / 25 / 40 | abuse limits for the public demo |
| `MRAG_TOP_K_TEXT` / `MRAG_TOP_K_IMAGES` | 6 / 3 | default context size (also adjustable in the UI) |
| `MRAG_LLM_BACKEND` | `transformers` | Keep this value on ZeroGPU. `extractive` is for local/CPU-only copies with no LLM. |

The Gradio event chain runs retrieval on CPU, then calls the **directly registered `@spaces.GPU` handler** for streaming, **only if retrieval succeeds**. CPU callbacks share a one-task queue, GPU generation has concurrency one, and a session cannot index or clear its files during an answer. On either generation success or failure, a web-process callback unlocks the controls and clears the pending prompt and image paths.

GPU time per answer is requested dynamically: 20 s, plus 4 s per image, plus 1 s per 12 tokens of the "Max new tokens" setting, capped at 120 s. This is an **unbenchmarked estimate**, not a measured runtime guarantee. Start with 256 output tokens if visitors hit a quota or duration error, then tune after representative hosted measurements. Lower requests get better queue priority.

The file count is cumulative across uploads and demo loading. PDFs are limited to their first 40 pages by default; the rest are not indexed. Upload size is checked at both the Gradio upload endpoint and indexing handler. The UI answers each question independently; the displayed transcript is not passed as conversational memory to retrieval or the model.

## Run locally

```bash
pip install -r requirements.txt
python app.py                                  # same Gradio app; uses CUDA/MPS if present, else CPU
```

The package also works without Gradio:
- CLI: `python -m mrag ingest docs/`, then `python -m mrag ask "..." --backend ollama|openai|transformers|extractive`
- Streamlit UI: `streamlit run mrag/streamlit_app.py`

All settings are in `mrag/config.py` and can be overridden with `MRAG_<FIELD>` environment variables.

## Tests

```bash
pip install -r requirements-dev.txt
pytest -q
```

The bundled test suite covers:
- loaders, OCR and retrieval on real models (correct top-1 text and images, plus negative queries that must return no image)
- re-ingest and the CLI
- the Ollama and OpenAI-compatible clients, against local servers that speak their wire protocols
- the transformers VLM path, using a tiny Qwen3-VL built on disk (same processor, image-token expansion, vision tower and streaming)
- this Gradio app driven over its API with `gradio_client`: demo documents, streamed answers, uploads, and isolation between sessions

To run the VLM and app tests against a real checkpoint, set `MRAG_VLM_ID=Qwen/Qwen3-VL-2B-Instruct`.

## Changes from the original Claude ZIP

See `REVIEW_AND_TEST_REPORT.md` for the latest corrections, measured validation results and remaining hosted checks. Full hosted inference still needs to be tested on an actual ZeroGPU Space.