dosemate / README.md
DoseMate
Fix HF Space colorTo metadata
4d0c7fc
|
Raw History Blame Contribute Delete
14.2 kB
---
title: DoseMate
emoji: πŸ’Š
colorFrom: green
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
---
<div align="center">
# 🩺 PharmaChat
**A multimodal AI medical assistant that answers medicine questions β€” from text or images.**
[![Python](https://img.shields.io/badge/Python-3.10+-3776AB?style=flat&logo=python&logoColor=white)](https://python.org)
[![FastAPI](https://img.shields.io/badge/FastAPI-Backend-009688?style=flat&logo=fastapi&logoColor=white)](https://fastapi.tiangolo.com)
[![React](https://img.shields.io/badge/React-Frontend-61DAFB?style=flat&logo=react&logoColor=black)](https://react.dev)
[![LangGraph](https://img.shields.io/badge/LangGraph-MultiAgent-FF6B35?style=flat)](https://langchain-ai.github.io/langgraph)
[![ChromaDB](https://img.shields.io/badge/ChromaDB-VectorDB-E85D4A?style=flat)](https://trychroma.com)
[![PostgreSQL](https://img.shields.io/badge/PostgreSQL-Database-4169E1?style=flat&logo=postgresql&logoColor=white)](https://postgresql.org)
*Type a question. Upload a photo. Get a medical answer instantly.*
</div>
---
## What Is This?
PharmaChat is a conversational medical assistant that lets you ask natural language questions about medicines β€” or simply **photograph a medicine box** and ask about it. The system identifies the medicine from the image, retrieves relevant information from a structured knowledge base, and delivers a clear, concise answer.
Built around a **4-agent LangGraph pipeline**, the system routes each request through specialized agents depending on whether the input is text or an image. All conversations are persisted in **PostgreSQL** and served through a **FastAPI** backend to a clean **React** chat interface.
---
## How It Works
```
Text Input β†’ Rewrite Agent ─┐
β”œβ†’ Retriever Agent β†’ Response Agent β†’ Answer
Image Input β†’ Vision Agent β”€β”€β”˜
```
1. **Vision Agent** *(image path only)* β€” uses Gemini to identify the medicine name from a photo of its packaging
2. **Rewrite Agent** *(text path only)* β€” rewrites the user's query to be more specific and retrieval-friendly, using conversation history to resolve references like "it" or "that medicine"
3. **Retriever Agent** β€” searches ChromaDB for the most relevant chunks from the medicine knowledge base
4. **Response Agent** β€” reads the retrieved content and generates a concise, conversational answer
---
## Features
### πŸ€– Multimodal Multi-Agent Pipeline
- **LangGraph workflow** β€” 4 specialized agents connected in a conditional stateful graph
- **Vision input** β€” photograph a medicine box or strip; Gemini identifies the medicine automatically
- **Query rewriting** β€” resolves follow-up references and improves retrieval accuracy before searching
- **Stateful context** β€” full conversation history passed to each agent for multi-turn awareness
### πŸ” Retrieval Pipeline
- **ChromaDB vector store** β€” medicine knowledge stored as dense vectors for semantic search
- **HuggingFace embeddings** β€” `all-MiniLM-L6-v2` converts text to vectors locally, no API cost
- **Structured JSON knowledge base** β€” medicines stored with name, uses, dosage, side effects, and interactions
- **Configurable top-k** β€” retrieves top 3 chunks per query for focused, concise answers
### πŸ’Ύ Persistent Conversation History
- **PostgreSQL database** β€” every message (text and image path) saved automatically
- **Auto-generated titles** β€” conversation titles generated by the LLM from the first message
- **Optimistic UI updates** β€” title appears instantly from the first few words, replaced by the AI title after response
- **Full history sidebar** β€” all past conversations listed, click any to restore it completely
- **Image persistence** β€” uploaded images saved to disk and served as static files; displayed inline in restored chats
### ⚑ FastAPI Backend
- **RESTful API** β€” clean endpoints for conversations, messages, and chat
- **SQLAlchemy ORM** β€” typed database models with cascade deletes
- **Static file serving** β€” uploaded images served directly from the backend
- **CORS configured** β€” ready for local development with the React frontend
### 🎨 React Frontend
- **Clean chat UI** β€” user bubbles on the right, assistant on the left
- **Image preview before sending** β€” attached image shown in the input bar
- **Inline image rendering** β€” images displayed inside chat bubbles
- **Right-click to delete** β€” context menu on any conversation in the sidebar
- **Typing indicator** β€” animated dots while the assistant is generating a response
---
## System Architecture
```
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ User (Browser) β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ React UI β”‚
β”‚ Vite + Axios β”‚
β”‚ Chat bubbles β”‚
β”‚ Sidebar historyβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚ HTTP (REST)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ FastAPI β”‚
β”‚ app.py β”‚
β”‚ /chat β”‚
β”‚ /conversationsβ”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ PostgreSQL β”‚ β”‚ LangGraph Graph β”‚
β”‚ database.py β”‚ β”‚ workflow.py β”‚
β”‚ Conversations β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚ Messages β”‚ β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Vision Agent β”‚ β”‚ Rewrite Agent β”‚
β”‚ Gemini Flash β”‚ β”‚ Groq / Llama β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚ β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Retriever Agent β”‚
β”‚ ChromaDB β”‚
β”‚ HuggingFace β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Response Agent β”‚
β”‚ Groq / Llama β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```
---
## Project Structure
```
.
β”œβ”€β”€ app.py # FastAPI backend β€” all API endpoints
β”œβ”€β”€ agents.py # Vision, rewrite, retriever, and response agents
β”œβ”€β”€ workflow.py # LangGraph pipeline (conditional 4-node graph)
β”œβ”€β”€ state.py # Shared TypedDict state
β”œβ”€β”€ prompt.py # System prompts and prompt builders
β”œβ”€β”€ database.py # SQLAlchemy models and PostgreSQL connection
β”œβ”€β”€ main.py # Entry point
β”œβ”€β”€ medicines.json # Structured medicine knowledge base
β”œβ”€β”€ notebook.ipynb # Knowledge base ingestion notebook
β”‚
β”œβ”€β”€ frontend/ # React + Vite frontend
β”‚ β”œβ”€β”€ src/
β”‚ β”‚ β”œβ”€β”€ App.jsx # Root component β€” conversation state
β”‚ β”‚ β”œβ”€β”€ api.js # Axios API client
β”‚ β”‚ └── components/
β”‚ β”‚ β”œβ”€β”€ Sidebar.jsx # Conversation list + new chat
β”‚ β”‚ β”œβ”€β”€ ChatWindow.jsx # Message bubbles + loading indicator
β”‚ β”‚ └── InputBar.jsx # Text input + image attachment
β”‚ └── ...
β”‚
β”‚
β”œβ”€β”€ .env.example # API key template
β”œβ”€β”€ requirements.txt # Python dependencies
└── README.md
```
---
## Quickstart
### Prerequisites
- Python 3.10+
- Node.js 18+
- PostgreSQL running locally
- A [Groq API key](https://console.groq.com) (free)
- A [Google Gemini API key](https://aistudio.google.com) (free)
### 1. Clone the repository
```bash
git clone https://github.com/your-username/pharmchat.git
cd pharmchat
```
### 2. Install Python dependencies
```bash
pip install -r requirements.txt
```
### 3. Set up environment variables
```bash
cp .env.example .env
```
Edit `.env`:
```env
GROQ_API_KEY="your_groq_key_here"
GOOGLE_API_KEY="your_gemini_key_here"
```
### 4. Set up PostgreSQL
Create a database and update the connection URL in `database.py`:
```python
Database_URL = "postgresql://postgres:yourpassword@localhost:5432/medicine_assistant"
```
Then create the tables by running the backend once β€” SQLAlchemy will auto-create them on startup.
### 5. Ingest the knowledge base
Open `notebook.ipynb` and run all cells. This loads `medicines.json`, splits it into chunks, embeds them with HuggingFace, and stores them in ChromaDB.
> **Re-run this whenever you update `medicines.json`.**
> Delete `chroma_db/` first to avoid duplicate entries.
### 6. Start the backend
```bash
uvicorn app:app --reload
```
Backend runs at [http://localhost:8000](http://localhost:8000).
### 7. Start the frontend
```bash
cd frontend
npm install
npm run dev
```
Frontend runs at [http://localhost:5173](http://localhost:5173).
---
## API Reference
| Method | Endpoint | Description |
|---|---|---|
| `GET` | `/health` | Health check |
| `POST` | `/conversation` | Create a new conversation |
| `GET` | `/conversations` | List all conversations |
| `DELETE` | `/conversations/{id}` | Delete a conversation and its images |
| `GET` | `/conversations/{id}/messages` | Load all messages for a conversation |
| `POST` | `/chat` | Send a message (text and/or image) |
### Chat request body
```json
{
"query": "What are the side effects of this medicine?",
"image_base64": "<base64-encoded image or null>",
"conversation_id": 3
}
```
---
## Database Schema
```sql
CREATE TABLE conversations (
id SERIAL PRIMARY KEY,
user_id INTEGER,
title VARCHAR(100),
created_at TIMESTAMP DEFAULT NOW()
);
CREATE TABLE messages (
id SERIAL PRIMARY KEY,
conversation_id INTEGER REFERENCES conversations(id) ON DELETE CASCADE,
role VARCHAR(10), -- 'user' or 'bot'
text TEXT,
image_path VARCHAR(255), -- URL to served image file
created_at TIMESTAMP DEFAULT NOW()
);
```
---
## Tech Stack
| Layer | Technology | Why |
|---|---|---|
| Agent Orchestration | LangGraph | Stateful multi-agent graph with conditional routing |
| LLM | Groq (Llama 3.1 8B) | Fast inference, generous free tier |
| Vision | Google Gemini 2.5 Flash Lite | Multimodal β€” identifies medicines from images |
| Embeddings | HuggingFace `all-MiniLM-L6-v2` | Free, runs locally, strong semantic quality |
| Vector DB | ChromaDB | Lightweight, local, zero infrastructure |
| Backend | FastAPI + SQLAlchemy | Async-ready, typed, auto-docs at `/docs` |
| Database | PostgreSQL | Reliable relational storage with cascade deletes |
| Frontend | React + Vite | Fast, component-based chat UI |
| HTTP Client | Axios | Clean promise-based API calls |
| Observability | LangSmith | Traces every LangGraph run end-to-end |
| Config | python-dotenv | Simple `.env` file management |
---
## Design Decisions
**Why conditional routing at the graph entry point?**
Text and image queries need completely different first steps. A conditional edge from `START` lets LangGraph branch cleanly β€” image inputs go straight to the vision agent, text inputs go to query rewriting β€” without any `if` statements inside the agents themselves.
**Why Gemini for vision instead of the main LLM?**
Groq's Llama models don't support image input. Gemini Flash Lite handles vision cheaply and accurately, so the vision agent uses Gemini just for identification, then hands off a text query to the rest of the pipeline.
**Why store image paths instead of base64 in the database?**
Base64-encoded images are 33% larger than the raw file and bloat the database quickly. Saving images to disk and storing only the URL keeps the database lean and makes images re-loadable in restored conversations.
**Why query rewriting before retrieval?**
User questions are often short and vague β€” "what are its side effects?" doesn't retrieve anything useful. The rewrite agent uses conversation history to resolve pronouns and expand the query into a full, specific sentence before it hits ChromaDB.
**Why optimistic title updates in the frontend?**
LLM title generation adds a noticeable delay before the sidebar updates. Showing the first 5 words of the user's message instantly makes the UI feel responsive, and the real AI-generated title replaces it silently after the response arrives.
---
## Environment Variables
| Variable | Required | Description |
|---|---|---|
| `GROQ_API_KEY` | βœ… | Groq API key for LLM inference |
| `GOOGLE_API_KEY` | βœ… | Google Gemini API key for vision |
| `LANGSMITH_API_KEY` | Optional | LangSmith tracing |
---
## License
MIT