---
title: DoseMate
emoji: π
colorFrom: green
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
---
# π©Ί PharmaChat
**A multimodal AI medical assistant that answers medicine questions β from text or images.**
[](https://python.org)
[](https://fastapi.tiangolo.com)
[](https://react.dev)
[](https://langchain-ai.github.io/langgraph)
[](https://trychroma.com)
[](https://postgresql.org)
*Type a question. Upload a photo. Get a medical answer instantly.*
---
## What Is This?
PharmaChat is a conversational medical assistant that lets you ask natural language questions about medicines β or simply **photograph a medicine box** and ask about it. The system identifies the medicine from the image, retrieves relevant information from a structured knowledge base, and delivers a clear, concise answer.
Built around a **4-agent LangGraph pipeline**, the system routes each request through specialized agents depending on whether the input is text or an image. All conversations are persisted in **PostgreSQL** and served through a **FastAPI** backend to a clean **React** chat interface.
---
## How It Works
```
Text Input β Rewrite Agent ββ
ββ Retriever Agent β Response Agent β Answer
Image Input β Vision Agent βββ
```
1. **Vision Agent** *(image path only)* β uses Gemini to identify the medicine name from a photo of its packaging
2. **Rewrite Agent** *(text path only)* β rewrites the user's query to be more specific and retrieval-friendly, using conversation history to resolve references like "it" or "that medicine"
3. **Retriever Agent** β searches ChromaDB for the most relevant chunks from the medicine knowledge base
4. **Response Agent** β reads the retrieved content and generates a concise, conversational answer
---
## Features
### π€ Multimodal Multi-Agent Pipeline
- **LangGraph workflow** β 4 specialized agents connected in a conditional stateful graph
- **Vision input** β photograph a medicine box or strip; Gemini identifies the medicine automatically
- **Query rewriting** β resolves follow-up references and improves retrieval accuracy before searching
- **Stateful context** β full conversation history passed to each agent for multi-turn awareness
### π Retrieval Pipeline
- **ChromaDB vector store** β medicine knowledge stored as dense vectors for semantic search
- **HuggingFace embeddings** β `all-MiniLM-L6-v2` converts text to vectors locally, no API cost
- **Structured JSON knowledge base** β medicines stored with name, uses, dosage, side effects, and interactions
- **Configurable top-k** β retrieves top 3 chunks per query for focused, concise answers
### πΎ Persistent Conversation History
- **PostgreSQL database** β every message (text and image path) saved automatically
- **Auto-generated titles** β conversation titles generated by the LLM from the first message
- **Optimistic UI updates** β title appears instantly from the first few words, replaced by the AI title after response
- **Full history sidebar** β all past conversations listed, click any to restore it completely
- **Image persistence** β uploaded images saved to disk and served as static files; displayed inline in restored chats
### β‘ FastAPI Backend
- **RESTful API** β clean endpoints for conversations, messages, and chat
- **SQLAlchemy ORM** β typed database models with cascade deletes
- **Static file serving** β uploaded images served directly from the backend
- **CORS configured** β ready for local development with the React frontend
### π¨ React Frontend
- **Clean chat UI** β user bubbles on the right, assistant on the left
- **Image preview before sending** β attached image shown in the input bar
- **Inline image rendering** β images displayed inside chat bubbles
- **Right-click to delete** β context menu on any conversation in the sidebar
- **Typing indicator** β animated dots while the assistant is generating a response
---
## System Architecture
```
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β User (Browser) β
βββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββ
β
ββββββββββΌβββββββββ
β React UI β
β Vite + Axios β
β Chat bubbles β
β Sidebar historyβ
ββββββββββ¬βββββββββ
β HTTP (REST)
ββββββββββΌβββββββββ
β FastAPI β
β app.py β
β /chat β
β /conversationsβ
ββββββββββ¬βββββββββ
β
βββββββββββββΌββββββββββββ
β β
ββββββββββΌβββββββββ βββββββββββΌβββββββββ
β PostgreSQL β β LangGraph Graph β
β database.py β β workflow.py β
β Conversations β βββββββββββ¬ββββββββββ
β Messages β β
βββββββββββββββββββ ββββββββββ΄βββββββββββββββββββββββ
β β
ββββββββββΌβββββββββ βββββββββββΌβββββββββ
β Vision Agent β β Rewrite Agent β
β Gemini Flash β β Groq / Llama β
ββββββββββ¬βββββββββ βββββββββββ¬βββββββββ
β β
ββββββββββββββββ¬βββββββββββββββββ
β
ββββββββββΌβββββββββ
β Retriever Agent β
β ChromaDB β
β HuggingFace β
ββββββββββ¬βββββββββ
β
ββββββββββΌβββββββββ
β Response Agent β
β Groq / Llama β
βββββββββββββββββββ
```
---
## Project Structure
```
.
βββ app.py # FastAPI backend β all API endpoints
βββ agents.py # Vision, rewrite, retriever, and response agents
βββ workflow.py # LangGraph pipeline (conditional 4-node graph)
βββ state.py # Shared TypedDict state
βββ prompt.py # System prompts and prompt builders
βββ database.py # SQLAlchemy models and PostgreSQL connection
βββ main.py # Entry point
βββ medicines.json # Structured medicine knowledge base
βββ notebook.ipynb # Knowledge base ingestion notebook
β
βββ frontend/ # React + Vite frontend
β βββ src/
β β βββ App.jsx # Root component β conversation state
β β βββ api.js # Axios API client
β β βββ components/
β β βββ Sidebar.jsx # Conversation list + new chat
β β βββ ChatWindow.jsx # Message bubbles + loading indicator
β β βββ InputBar.jsx # Text input + image attachment
β βββ ...
β
β
βββ .env.example # API key template
βββ requirements.txt # Python dependencies
βββ README.md
```
---
## Quickstart
### Prerequisites
- Python 3.10+
- Node.js 18+
- PostgreSQL running locally
- A [Groq API key](https://console.groq.com) (free)
- A [Google Gemini API key](https://aistudio.google.com) (free)
### 1. Clone the repository
```bash
git clone https://github.com/your-username/pharmchat.git
cd pharmchat
```
### 2. Install Python dependencies
```bash
pip install -r requirements.txt
```
### 3. Set up environment variables
```bash
cp .env.example .env
```
Edit `.env`:
```env
GROQ_API_KEY="your_groq_key_here"
GOOGLE_API_KEY="your_gemini_key_here"
```
### 4. Set up PostgreSQL
Create a database and update the connection URL in `database.py`:
```python
Database_URL = "postgresql://postgres:yourpassword@localhost:5432/medicine_assistant"
```
Then create the tables by running the backend once β SQLAlchemy will auto-create them on startup.
### 5. Ingest the knowledge base
Open `notebook.ipynb` and run all cells. This loads `medicines.json`, splits it into chunks, embeds them with HuggingFace, and stores them in ChromaDB.
> **Re-run this whenever you update `medicines.json`.**
> Delete `chroma_db/` first to avoid duplicate entries.
### 6. Start the backend
```bash
uvicorn app:app --reload
```
Backend runs at [http://localhost:8000](http://localhost:8000).
### 7. Start the frontend
```bash
cd frontend
npm install
npm run dev
```
Frontend runs at [http://localhost:5173](http://localhost:5173).
---
## API Reference
| Method | Endpoint | Description |
|---|---|---|
| `GET` | `/health` | Health check |
| `POST` | `/conversation` | Create a new conversation |
| `GET` | `/conversations` | List all conversations |
| `DELETE` | `/conversations/{id}` | Delete a conversation and its images |
| `GET` | `/conversations/{id}/messages` | Load all messages for a conversation |
| `POST` | `/chat` | Send a message (text and/or image) |
### Chat request body
```json
{
"query": "What are the side effects of this medicine?",
"image_base64": "",
"conversation_id": 3
}
```
---
## Database Schema
```sql
CREATE TABLE conversations (
id SERIAL PRIMARY KEY,
user_id INTEGER,
title VARCHAR(100),
created_at TIMESTAMP DEFAULT NOW()
);
CREATE TABLE messages (
id SERIAL PRIMARY KEY,
conversation_id INTEGER REFERENCES conversations(id) ON DELETE CASCADE,
role VARCHAR(10), -- 'user' or 'bot'
text TEXT,
image_path VARCHAR(255), -- URL to served image file
created_at TIMESTAMP DEFAULT NOW()
);
```
---
## Tech Stack
| Layer | Technology | Why |
|---|---|---|
| Agent Orchestration | LangGraph | Stateful multi-agent graph with conditional routing |
| LLM | Groq (Llama 3.1 8B) | Fast inference, generous free tier |
| Vision | Google Gemini 2.5 Flash Lite | Multimodal β identifies medicines from images |
| Embeddings | HuggingFace `all-MiniLM-L6-v2` | Free, runs locally, strong semantic quality |
| Vector DB | ChromaDB | Lightweight, local, zero infrastructure |
| Backend | FastAPI + SQLAlchemy | Async-ready, typed, auto-docs at `/docs` |
| Database | PostgreSQL | Reliable relational storage with cascade deletes |
| Frontend | React + Vite | Fast, component-based chat UI |
| HTTP Client | Axios | Clean promise-based API calls |
| Observability | LangSmith | Traces every LangGraph run end-to-end |
| Config | python-dotenv | Simple `.env` file management |
---
## Design Decisions
**Why conditional routing at the graph entry point?**
Text and image queries need completely different first steps. A conditional edge from `START` lets LangGraph branch cleanly β image inputs go straight to the vision agent, text inputs go to query rewriting β without any `if` statements inside the agents themselves.
**Why Gemini for vision instead of the main LLM?**
Groq's Llama models don't support image input. Gemini Flash Lite handles vision cheaply and accurately, so the vision agent uses Gemini just for identification, then hands off a text query to the rest of the pipeline.
**Why store image paths instead of base64 in the database?**
Base64-encoded images are 33% larger than the raw file and bloat the database quickly. Saving images to disk and storing only the URL keeps the database lean and makes images re-loadable in restored conversations.
**Why query rewriting before retrieval?**
User questions are often short and vague β "what are its side effects?" doesn't retrieve anything useful. The rewrite agent uses conversation history to resolve pronouns and expand the query into a full, specific sentence before it hits ChromaDB.
**Why optimistic title updates in the frontend?**
LLM title generation adds a noticeable delay before the sidebar updates. Showing the first 5 words of the user's message instantly makes the UI feel responsive, and the real AI-generated title replaces it silently after the response arrives.
---
## Environment Variables
| Variable | Required | Description |
|---|---|---|
| `GROQ_API_KEY` | β
| Groq API key for LLM inference |
| `GOOGLE_API_KEY` | β
| Google Gemini API key for vision |
| `LANGSMITH_API_KEY` | Optional | LangSmith tracing |
---
## License
MIT