Spaces:
Sleeping
Sleeping
fix gpu
Browse files- README.md +4 -3
- project/.env.example +1 -1
- project/README.md +26 -15
- project/db/dense_embeddings.py +34 -0
- project/db/vector_db_manager.py +21 -8
- project/requirements.txt +1 -12
- requirements.txt +1 -0
README.md
CHANGED
|
@@ -10,8 +10,9 @@ app_file: project/app.py
|
|
| 10 |
---
|
| 11 |
|
| 12 |
**Deployable app:** The Python/Gradio app now uses Groq (`GROQ_API_KEY`) for chat
|
| 13 |
-
and
|
| 14 |
-
|
|
|
|
| 15 |
Ollama examples. See [deployment instructions](project/README.md#hugging-face-spaces-recommended)
|
| 16 |
for Hugging Face Spaces, Render, secrets, and persistent storage.
|
| 17 |
|
|
@@ -1402,7 +1403,7 @@ The chat interface will appear at the end.
|
|
| 1402 |
# Run from the project root.
|
| 1403 |
# Windows PowerShell: install only the deployable app
|
| 1404 |
uv venv --python 3.13 .venv
|
| 1405 |
-
uv pip install --python .venv\Scripts\python.exe --torch-backend cpu -r
|
| 1406 |
Copy-Item project\.env.example project\.env
|
| 1407 |
# Edit project\.env and set GROQ_API_KEY.
|
| 1408 |
# For notebook/evaluation dependencies too, install requirements-notebooks.txt instead.
|
|
|
|
| 10 |
---
|
| 11 |
|
| 12 |
**Deployable app:** The Python/Gradio app now uses Groq (`GROQ_API_KEY`) for chat
|
| 13 |
+
and Hugging Face embeddings for retrieval (ZeroGPU on Spaces, CPU locally).
|
| 14 |
+
Use `uv` to install the runtime
|
| 15 |
+
dependencies in the root `requirements.txt`. The learning notebooks retain their
|
| 16 |
Ollama examples. See [deployment instructions](project/README.md#hugging-face-spaces-recommended)
|
| 17 |
for Hugging Face Spaces, Render, secrets, and persistent storage.
|
| 18 |
|
|
|
|
| 1403 |
# Run from the project root.
|
| 1404 |
# Windows PowerShell: install only the deployable app
|
| 1405 |
uv venv --python 3.13 .venv
|
| 1406 |
+
uv pip install --python .venv\Scripts\python.exe --torch-backend cpu -r requirements.txt
|
| 1407 |
Copy-Item project\.env.example project\.env
|
| 1408 |
# Edit project\.env and set GROQ_API_KEY.
|
| 1409 |
# For notebook/evaluation dependencies too, install requirements-notebooks.txt instead.
|
project/.env.example
CHANGED
|
@@ -3,7 +3,7 @@
|
|
| 3 |
GROQ_API_KEY=
|
| 4 |
GROQ_MODEL=llama-3.3-70b-versatile
|
| 5 |
|
| 6 |
-
# MiniLM
|
| 7 |
# Changing embeddings requires clearing and re-indexing existing documents.
|
| 8 |
# DENSE_MODEL=Qwen/Qwen3-Embedding-0.6B
|
| 9 |
|
|
|
|
| 3 |
GROQ_API_KEY=
|
| 4 |
GROQ_MODEL=llama-3.3-70b-versatile
|
| 5 |
|
| 6 |
+
# MiniLM embeddings use ZeroGPU on HF and CPU locally.
|
| 7 |
# Changing embeddings requires clearing and re-indexing existing documents.
|
| 8 |
# DENSE_MODEL=Qwen/Qwen3-Embedding-0.6B
|
| 9 |
|
project/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# Document Chat Documentation
|
| 2 |
|
| 3 |
-
An **Agentic Retrieval-Augmented Generation (RAG)** system built with **LangGraph**, featuring **parent–child chunking**, **hybrid dense + sparse retrieval**, and Groq-powered chat with a Gradio frontend. Embeddings
|
| 4 |
|
| 5 |
|
| 6 |
## Table of Contents
|
|
@@ -17,15 +17,17 @@ From the repository root, use `uv` (Windows PowerShell):
|
|
| 17 |
|
| 18 |
```powershell
|
| 19 |
uv venv --python 3.13 .venv
|
| 20 |
-
uv pip install --python .venv\Scripts\python.exe --torch-backend cpu -r
|
| 21 |
Copy-Item project\.env.example project\.env
|
| 22 |
```
|
| 23 |
|
| 24 |
Set `GROQ_API_KEY` in `project/.env`. Get a key from
|
| 25 |
https://console.groq.com/keys. Never commit `.env` or expose the key in frontend
|
| 26 |
-
code. The root `requirements.txt`
|
| 27 |
-
|
| 28 |
-
|
|
|
|
|
|
|
| 29 |
|
| 30 |
### Running the Application
|
| 31 |
|
|
@@ -96,7 +98,8 @@ PDF → Markdown Conversion → Parent/Child Chunking → Vector Indexing → Ag
|
|
| 96 |
|
| 97 |
| File | Purpose |
|
| 98 |
|------|---------|
|
| 99 |
-
| `project/db/vector_db_manager.py` | Qdrant client wrapper with
|
|
|
|
| 100 |
| `project/db/parent_store_manager.py` | File-backed storage for parent chunks |
|
| 101 |
|
| 102 |
### RAG Agent (LangGraph)
|
|
@@ -379,7 +382,7 @@ ACTIVE_LLM_CONFIG = "google" # Switch to Gemini Pro
|
|
| 379 |
**Step 1:** Update `project/config.py`
|
| 380 |
|
| 381 |
```python
|
| 382 |
-
# Default:
|
| 383 |
DENSE_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
|
| 384 |
|
| 385 |
# Example: Alternative Sentence Transformers model
|
|
@@ -405,9 +408,9 @@ from langchain_huggingface import HuggingFaceEmbeddings
|
|
| 405 |
from langchain_qdrant import FastEmbedSparse
|
| 406 |
import config
|
| 407 |
|
| 408 |
-
|
| 409 |
-
|
| 410 |
-
)
|
| 411 |
self.__sparse_embeddings = FastEmbedSparse(model_name=config.SPARSE_MODEL)
|
| 412 |
```
|
| 413 |
|
|
@@ -415,7 +418,7 @@ self.__sparse_embeddings = FastEmbedSparse(model_name=config.SPARSE_MODEL)
|
|
| 415 |
|
| 416 |
| Model | Context Size | Vector Dimension | Speed | Quality | Use Case |
|
| 417 |
|-------|--------------|------------------|-------|---------|----------|
|
| 418 |
-
| all-MiniLM-L6-v2 | 256 tokens | 384 | Fast | Good | Default
|
| 419 |
| all-mpnet-base-v2 | 512 tokens | 768 | Medium | Excellent | High-accuracy semantic search |
|
| 420 |
| bge-large-en-v1.5 | 512 tokens | 1024 | Slow | Best | Production-grade retrieval on GPU |
|
| 421 |
| google/embeddinggemma-300m | 2048 tokens | 768 | Fast | Very Good | Lightweight, efficient multilingual retrieval |
|
|
@@ -663,10 +666,18 @@ Once running, open `http://localhost:7860`.
|
|
| 663 |
Upload a small Markdown/PDF file under **Documents**, click **Add Documents**,
|
| 664 |
and ask a question under **Chat**.
|
| 665 |
|
| 666 |
-
The app
|
| 667 |
-
|
| 668 |
-
|
| 669 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 670 |
|
| 671 |
The UI is branded **Document Chat**, with a teal-accented header, a compact
|
| 672 |
chat panel, and the same **Documents** and **Chat** workflow.
|
|
|
|
| 1 |
# Document Chat Documentation
|
| 2 |
|
| 3 |
+
An **Agentic Retrieval-Augmented Generation (RAG)** system built with **LangGraph**, featuring **parent–child chunking**, **hybrid dense + sparse retrieval**, and Groq-powered chat with a Gradio frontend. Embeddings use ZeroGPU on HF Spaces and CPU locally; no Ollama server is required. The learning notebooks retain their Ollama setup.
|
| 4 |
|
| 5 |
|
| 6 |
## Table of Contents
|
|
|
|
| 17 |
|
| 18 |
```powershell
|
| 19 |
uv venv --python 3.13 .venv
|
| 20 |
+
uv pip install --python .venv\Scripts\python.exe --torch-backend cpu -r requirements.txt
|
| 21 |
Copy-Item project\.env.example project\.env
|
| 22 |
```
|
| 23 |
|
| 24 |
Set `GROQ_API_KEY` in `project/.env`. Get a key from
|
| 25 |
https://console.groq.com/keys. Never commit `.env` or expose the key in frontend
|
| 26 |
+
code. The root `requirements.txt` is the standalone runtime dependency manifest.
|
| 27 |
+
HF Gradio mounts only this file during installation, so it must not include
|
| 28 |
+
other local requirement files. Notebook/evaluation tools are in
|
| 29 |
+
`requirements-notebooks.txt`. The optional `project/requirements.txt` redirects
|
| 30 |
+
local installs to the root manifest; Docker installs the root file directly.
|
| 31 |
|
| 32 |
### Running the Application
|
| 33 |
|
|
|
|
| 98 |
|
| 99 |
| File | Purpose |
|
| 100 |
|------|---------|
|
| 101 |
+
| `project/db/vector_db_manager.py` | Qdrant client wrapper with inference-free schema validation |
|
| 102 |
+
| `project/db/dense_embeddings.py` | MiniLM embeddings with ZeroGPU inference on Spaces and CPU inference locally |
|
| 103 |
| `project/db/parent_store_manager.py` | File-backed storage for parent chunks |
|
| 104 |
|
| 105 |
### RAG Agent (LangGraph)
|
|
|
|
| 382 |
**Step 1:** Update `project/config.py`
|
| 383 |
|
| 384 |
```python
|
| 385 |
+
# Default: Compact model (ZeroGPU on HF, CPU locally)
|
| 386 |
DENSE_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
|
| 387 |
|
| 388 |
# Example: Alternative Sentence Transformers model
|
|
|
|
| 408 |
from langchain_qdrant import FastEmbedSparse
|
| 409 |
import config
|
| 410 |
|
| 411 |
+
from db.dense_embeddings import DenseEmbeddings
|
| 412 |
+
|
| 413 |
+
self.__dense_embeddings = DenseEmbeddings(model_name=config.DENSE_MODEL)
|
| 414 |
self.__sparse_embeddings = FastEmbedSparse(model_name=config.SPARSE_MODEL)
|
| 415 |
```
|
| 416 |
|
|
|
|
| 418 |
|
| 419 |
| Model | Context Size | Vector Dimension | Speed | Quality | Use Case |
|
| 420 |
|-------|--------------|------------------|-------|---------|----------|
|
| 421 |
+
| all-MiniLM-L6-v2 | 256 tokens | 384 | Fast | Good | Default compact embedding model |
|
| 422 |
| all-mpnet-base-v2 | 512 tokens | 768 | Medium | Excellent | High-accuracy semantic search |
|
| 423 |
| bge-large-en-v1.5 | 512 tokens | 1024 | Slow | Best | Production-grade retrieval on GPU |
|
| 424 |
| google/embeddinggemma-300m | 2048 tokens | 768 | Fast | Very Good | Lightweight, efficient multilingual retrieval |
|
|
|
|
| 666 |
Upload a small Markdown/PDF file under **Documents**, click **Add Documents**,
|
| 667 |
and ask a question under **Chat**.
|
| 668 |
|
| 669 |
+
The app loads MiniLM on **CUDA** at startup when HF sets `SPACES_ZERO_GPU=true`.
|
| 670 |
+
Dense embedding inference runs inside `@spaces.GPU(duration=30)`, both during
|
| 671 |
+
document indexing and query retrieval. Only texts and embedding results cross
|
| 672 |
+
the worker boundary; local Qdrant writes and Groq calls stay outside GPU tasks.
|
| 673 |
+
Startup obtains the vector dimension from model metadata and validates the
|
| 674 |
+
Qdrant schema without making an inference request.
|
| 675 |
+
|
| 676 |
+
ZeroGPU requires a registered GPU function during startup; a CPU-only app is
|
| 677 |
+
not sufficient on this hardware tier. Embedding requests consume the visitor's
|
| 678 |
+
GPU quota and may queue. Large indexing jobs can require multiple allocations.
|
| 679 |
+
CPU embedding inference is retained for local and non-ZeroGPU deployments.
|
| 680 |
+
Verify first startup and indexing in the Space logs.
|
| 681 |
|
| 682 |
The UI is branded **Document Chat**, with a teal-accented header, a compact
|
| 683 |
chat panel, and the same **Documents** and **Chat** workflow.
|
project/db/dense_embeddings.py
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import os
|
| 2 |
+
import spaces
|
| 3 |
+
from langchain_core.embeddings import Embeddings
|
| 4 |
+
from sentence_transformers import SentenceTransformer
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
class DenseEmbeddings(Embeddings):
|
| 8 |
+
def __init__(self, model_name: str):
|
| 9 |
+
is_zero_gpu = os.environ.get("SPACES_ZERO_GPU", "").lower() in {"1", "true"}
|
| 10 |
+
model = SentenceTransformer(
|
| 11 |
+
model_name,
|
| 12 |
+
device="cuda" if is_zero_gpu else "cpu",
|
| 13 |
+
)
|
| 14 |
+
dimension = model.get_sentence_embedding_dimension()
|
| 15 |
+
if not isinstance(dimension, int) or dimension <= 0:
|
| 16 |
+
raise ValueError(f"Cannot determine embedding dimension for '{model_name}'.")
|
| 17 |
+
self.vector_size = dimension
|
| 18 |
+
|
| 19 |
+
# Capture the model in the worker, rather than sending CUDA weights as arguments.
|
| 20 |
+
@spaces.GPU(duration=30)
|
| 21 |
+
def encode(texts: list[str]) -> list[list[float]]:
|
| 22 |
+
return model.encode(
|
| 23 |
+
texts,
|
| 24 |
+
show_progress_bar=False,
|
| 25 |
+
convert_to_numpy=True,
|
| 26 |
+
).tolist()
|
| 27 |
+
|
| 28 |
+
self._encode = encode
|
| 29 |
+
|
| 30 |
+
def embed_documents(self, texts: list[str]) -> list[list[float]]:
|
| 31 |
+
return self._encode([text.replace("\n", " ") for text in texts])
|
| 32 |
+
|
| 33 |
+
def embed_query(self, text: str) -> list[float]:
|
| 34 |
+
return self.embed_documents([text])[0]
|
project/db/vector_db_manager.py
CHANGED
|
@@ -1,23 +1,20 @@
|
|
| 1 |
import config
|
| 2 |
-
from
|
| 3 |
from langchain_qdrant import QdrantVectorStore, FastEmbedSparse, RetrievalMode
|
| 4 |
from qdrant_client import QdrantClient
|
| 5 |
from qdrant_client.http import models as qmodels
|
| 6 |
|
| 7 |
class VectorDbManager:
|
| 8 |
__client: QdrantClient
|
| 9 |
-
__dense_embeddings:
|
| 10 |
__sparse_embeddings: FastEmbedSparse
|
| 11 |
def __init__(self):
|
| 12 |
self.__client = QdrantClient(path=config.QDRANT_DB_PATH)
|
| 13 |
-
self.__dense_embeddings =
|
| 14 |
-
model_name=config.DENSE_MODEL,
|
| 15 |
-
model_kwargs={"device": "cpu"},
|
| 16 |
-
)
|
| 17 |
self.__sparse_embeddings = FastEmbedSparse(model_name=config.SPARSE_MODEL)
|
| 18 |
|
| 19 |
def _dense_vector_size(self):
|
| 20 |
-
return
|
| 21 |
|
| 22 |
@staticmethod
|
| 23 |
def _collection_vector_size(collection_info):
|
|
@@ -41,6 +38,20 @@ class VectorDbManager:
|
|
| 41 |
print(f"✓ Collection created: {collection_name}")
|
| 42 |
else:
|
| 43 |
collection_info = self.__client.get_collection(collection_name)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
existing_size = self._collection_vector_size(collection_info)
|
| 45 |
if existing_size and existing_size != expected_size:
|
| 46 |
raise ValueError(
|
|
@@ -67,7 +78,9 @@ class VectorDbManager:
|
|
| 67 |
embedding=self.__dense_embeddings,
|
| 68 |
sparse_embedding=self.__sparse_embeddings,
|
| 69 |
retrieval_mode=RetrievalMode.HYBRID,
|
| 70 |
-
sparse_vector_name=config.SPARSE_VECTOR_NAME
|
|
|
|
|
|
|
| 71 |
)
|
| 72 |
except Exception as e:
|
| 73 |
raise RuntimeError(f"Unable to initialize Qdrant collection '{collection_name}'.") from e
|
|
|
|
| 1 |
import config
|
| 2 |
+
from db.dense_embeddings import DenseEmbeddings
|
| 3 |
from langchain_qdrant import QdrantVectorStore, FastEmbedSparse, RetrievalMode
|
| 4 |
from qdrant_client import QdrantClient
|
| 5 |
from qdrant_client.http import models as qmodels
|
| 6 |
|
| 7 |
class VectorDbManager:
|
| 8 |
__client: QdrantClient
|
| 9 |
+
__dense_embeddings: DenseEmbeddings
|
| 10 |
__sparse_embeddings: FastEmbedSparse
|
| 11 |
def __init__(self):
|
| 12 |
self.__client = QdrantClient(path=config.QDRANT_DB_PATH)
|
| 13 |
+
self.__dense_embeddings = DenseEmbeddings(model_name=config.DENSE_MODEL)
|
|
|
|
|
|
|
|
|
|
| 14 |
self.__sparse_embeddings = FastEmbedSparse(model_name=config.SPARSE_MODEL)
|
| 15 |
|
| 16 |
def _dense_vector_size(self):
|
| 17 |
+
return self.__dense_embeddings.vector_size
|
| 18 |
|
| 19 |
@staticmethod
|
| 20 |
def _collection_vector_size(collection_info):
|
|
|
|
| 38 |
print(f"✓ Collection created: {collection_name}")
|
| 39 |
else:
|
| 40 |
collection_info = self.__client.get_collection(collection_name)
|
| 41 |
+
vectors_config = collection_info.config.params.vectors
|
| 42 |
+
dense_config = (
|
| 43 |
+
vectors_config.get("") if isinstance(vectors_config, dict)
|
| 44 |
+
else vectors_config
|
| 45 |
+
)
|
| 46 |
+
if dense_config is None:
|
| 47 |
+
raise ValueError(f"Qdrant collection '{collection_name}' must have an unnamed dense vector.")
|
| 48 |
+
if dense_config.distance != qmodels.Distance.COSINE:
|
| 49 |
+
raise ValueError(f"Qdrant collection '{collection_name}' must use cosine distance.")
|
| 50 |
+
sparse_config = collection_info.config.params.sparse_vectors or {}
|
| 51 |
+
if config.SPARSE_VECTOR_NAME not in sparse_config:
|
| 52 |
+
raise ValueError(
|
| 53 |
+
f"Qdrant collection '{collection_name}' is missing sparse vector '{config.SPARSE_VECTOR_NAME}'."
|
| 54 |
+
)
|
| 55 |
existing_size = self._collection_vector_size(collection_info)
|
| 56 |
if existing_size and existing_size != expected_size:
|
| 57 |
raise ValueError(
|
|
|
|
| 78 |
embedding=self.__dense_embeddings,
|
| 79 |
sparse_embedding=self.__sparse_embeddings,
|
| 80 |
retrieval_mode=RetrievalMode.HYBRID,
|
| 81 |
+
sparse_vector_name=config.SPARSE_VECTOR_NAME,
|
| 82 |
+
# Schema is checked above without requesting GPU inference at startup.
|
| 83 |
+
validate_collection_config=False,
|
| 84 |
)
|
| 85 |
except Exception as e:
|
| 86 |
raise RuntimeError(f"Unable to initialize Qdrant collection '{collection_name}'.") from e
|
project/requirements.txt
CHANGED
|
@@ -1,12 +1 @@
|
|
| 1 |
-
|
| 2 |
-
gradio==6.29.0
|
| 3 |
-
langchain-groq==1.1.3
|
| 4 |
-
langchain-huggingface==1.2.2
|
| 5 |
-
langchain-qdrant==1.1.0
|
| 6 |
-
langchain-text-splitters==1.1.2
|
| 7 |
-
langfuse==4.16.0
|
| 8 |
-
langgraph==1.2.12
|
| 9 |
-
pymupdf4llm==1.28.2
|
| 10 |
-
python-dotenv==1.2.4
|
| 11 |
-
sentence-transformers==6.1.0
|
| 12 |
-
tiktoken==0.14.0
|
|
|
|
| 1 |
+
-r ../requirements.txt
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
requirements.txt
CHANGED
|
@@ -9,4 +9,5 @@ langgraph==1.2.12
|
|
| 9 |
pymupdf4llm==1.28.2
|
| 10 |
python-dotenv==1.2.4
|
| 11 |
sentence-transformers==6.1.0
|
|
|
|
| 12 |
tiktoken==0.14.0
|
|
|
|
| 9 |
pymupdf4llm==1.28.2
|
| 10 |
python-dotenv==1.2.4
|
| 11 |
sentence-transformers==6.1.0
|
| 12 |
+
spaces==0.51.3
|
| 13 |
tiktoken==0.14.0
|