--- title: DoseMate emoji: πŸ’Š colorFrom: green colorTo: blue sdk: docker app_port: 7860 pinned: false ---
# 🩺 PharmaChat **A multimodal AI medical assistant that answers medicine questions β€” from text or images.** [![Python](https://img.shields.io/badge/Python-3.10+-3776AB?style=flat&logo=python&logoColor=white)](https://python.org) [![FastAPI](https://img.shields.io/badge/FastAPI-Backend-009688?style=flat&logo=fastapi&logoColor=white)](https://fastapi.tiangolo.com) [![React](https://img.shields.io/badge/React-Frontend-61DAFB?style=flat&logo=react&logoColor=black)](https://react.dev) [![LangGraph](https://img.shields.io/badge/LangGraph-MultiAgent-FF6B35?style=flat)](https://langchain-ai.github.io/langgraph) [![ChromaDB](https://img.shields.io/badge/ChromaDB-VectorDB-E85D4A?style=flat)](https://trychroma.com) [![PostgreSQL](https://img.shields.io/badge/PostgreSQL-Database-4169E1?style=flat&logo=postgresql&logoColor=white)](https://postgresql.org) *Type a question. Upload a photo. Get a medical answer instantly.*
--- ## What Is This? PharmaChat is a conversational medical assistant that lets you ask natural language questions about medicines β€” or simply **photograph a medicine box** and ask about it. The system identifies the medicine from the image, retrieves relevant information from a structured knowledge base, and delivers a clear, concise answer. Built around a **4-agent LangGraph pipeline**, the system routes each request through specialized agents depending on whether the input is text or an image. All conversations are persisted in **PostgreSQL** and served through a **FastAPI** backend to a clean **React** chat interface. --- ## How It Works ``` Text Input β†’ Rewrite Agent ─┐ β”œβ†’ Retriever Agent β†’ Response Agent β†’ Answer Image Input β†’ Vision Agent β”€β”€β”˜ ``` 1. **Vision Agent** *(image path only)* β€” uses Gemini to identify the medicine name from a photo of its packaging 2. **Rewrite Agent** *(text path only)* β€” rewrites the user's query to be more specific and retrieval-friendly, using conversation history to resolve references like "it" or "that medicine" 3. **Retriever Agent** β€” searches ChromaDB for the most relevant chunks from the medicine knowledge base 4. **Response Agent** β€” reads the retrieved content and generates a concise, conversational answer --- ## Features ### πŸ€– Multimodal Multi-Agent Pipeline - **LangGraph workflow** β€” 4 specialized agents connected in a conditional stateful graph - **Vision input** β€” photograph a medicine box or strip; Gemini identifies the medicine automatically - **Query rewriting** β€” resolves follow-up references and improves retrieval accuracy before searching - **Stateful context** β€” full conversation history passed to each agent for multi-turn awareness ### πŸ” Retrieval Pipeline - **ChromaDB vector store** β€” medicine knowledge stored as dense vectors for semantic search - **HuggingFace embeddings** β€” `all-MiniLM-L6-v2` converts text to vectors locally, no API cost - **Structured JSON knowledge base** β€” medicines stored with name, uses, dosage, side effects, and interactions - **Configurable top-k** β€” retrieves top 3 chunks per query for focused, concise answers ### πŸ’Ύ Persistent Conversation History - **PostgreSQL database** β€” every message (text and image path) saved automatically - **Auto-generated titles** β€” conversation titles generated by the LLM from the first message - **Optimistic UI updates** β€” title appears instantly from the first few words, replaced by the AI title after response - **Full history sidebar** β€” all past conversations listed, click any to restore it completely - **Image persistence** β€” uploaded images saved to disk and served as static files; displayed inline in restored chats ### ⚑ FastAPI Backend - **RESTful API** β€” clean endpoints for conversations, messages, and chat - **SQLAlchemy ORM** β€” typed database models with cascade deletes - **Static file serving** β€” uploaded images served directly from the backend - **CORS configured** β€” ready for local development with the React frontend ### 🎨 React Frontend - **Clean chat UI** β€” user bubbles on the right, assistant on the left - **Image preview before sending** β€” attached image shown in the input bar - **Inline image rendering** β€” images displayed inside chat bubbles - **Right-click to delete** β€” context menu on any conversation in the sidebar - **Typing indicator** β€” animated dots while the assistant is generating a response --- ## System Architecture ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ User (Browser) β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ React UI β”‚ β”‚ Vite + Axios β”‚ β”‚ Chat bubbles β”‚ β”‚ Sidebar historyβ”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ HTTP (REST) β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ FastAPI β”‚ β”‚ app.py β”‚ β”‚ /chat β”‚ β”‚ /conversationsβ”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ PostgreSQL β”‚ β”‚ LangGraph Graph β”‚ β”‚ database.py β”‚ β”‚ workflow.py β”‚ β”‚ Conversations β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ Messages β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Vision Agent β”‚ β”‚ Rewrite Agent β”‚ β”‚ Gemini Flash β”‚ β”‚ Groq / Llama β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Retriever Agent β”‚ β”‚ ChromaDB β”‚ β”‚ HuggingFace β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Response Agent β”‚ β”‚ Groq / Llama β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` --- ## Project Structure ``` . β”œβ”€β”€ app.py # FastAPI backend β€” all API endpoints β”œβ”€β”€ agents.py # Vision, rewrite, retriever, and response agents β”œβ”€β”€ workflow.py # LangGraph pipeline (conditional 4-node graph) β”œβ”€β”€ state.py # Shared TypedDict state β”œβ”€β”€ prompt.py # System prompts and prompt builders β”œβ”€β”€ database.py # SQLAlchemy models and PostgreSQL connection β”œβ”€β”€ main.py # Entry point β”œβ”€β”€ medicines.json # Structured medicine knowledge base β”œβ”€β”€ notebook.ipynb # Knowledge base ingestion notebook β”‚ β”œβ”€β”€ frontend/ # React + Vite frontend β”‚ β”œβ”€β”€ src/ β”‚ β”‚ β”œβ”€β”€ App.jsx # Root component β€” conversation state β”‚ β”‚ β”œβ”€β”€ api.js # Axios API client β”‚ β”‚ └── components/ β”‚ β”‚ β”œβ”€β”€ Sidebar.jsx # Conversation list + new chat β”‚ β”‚ β”œβ”€β”€ ChatWindow.jsx # Message bubbles + loading indicator β”‚ β”‚ └── InputBar.jsx # Text input + image attachment β”‚ └── ... β”‚ β”‚ β”œβ”€β”€ .env.example # API key template β”œβ”€β”€ requirements.txt # Python dependencies └── README.md ``` --- ## Quickstart ### Prerequisites - Python 3.10+ - Node.js 18+ - PostgreSQL running locally - A [Groq API key](https://console.groq.com) (free) - A [Google Gemini API key](https://aistudio.google.com) (free) ### 1. Clone the repository ```bash git clone https://github.com/your-username/pharmchat.git cd pharmchat ``` ### 2. Install Python dependencies ```bash pip install -r requirements.txt ``` ### 3. Set up environment variables ```bash cp .env.example .env ``` Edit `.env`: ```env GROQ_API_KEY="your_groq_key_here" GOOGLE_API_KEY="your_gemini_key_here" ``` ### 4. Set up PostgreSQL Create a database and update the connection URL in `database.py`: ```python Database_URL = "postgresql://postgres:yourpassword@localhost:5432/medicine_assistant" ``` Then create the tables by running the backend once β€” SQLAlchemy will auto-create them on startup. ### 5. Ingest the knowledge base Open `notebook.ipynb` and run all cells. This loads `medicines.json`, splits it into chunks, embeds them with HuggingFace, and stores them in ChromaDB. > **Re-run this whenever you update `medicines.json`.** > Delete `chroma_db/` first to avoid duplicate entries. ### 6. Start the backend ```bash uvicorn app:app --reload ``` Backend runs at [http://localhost:8000](http://localhost:8000). ### 7. Start the frontend ```bash cd frontend npm install npm run dev ``` Frontend runs at [http://localhost:5173](http://localhost:5173). --- ## API Reference | Method | Endpoint | Description | |---|---|---| | `GET` | `/health` | Health check | | `POST` | `/conversation` | Create a new conversation | | `GET` | `/conversations` | List all conversations | | `DELETE` | `/conversations/{id}` | Delete a conversation and its images | | `GET` | `/conversations/{id}/messages` | Load all messages for a conversation | | `POST` | `/chat` | Send a message (text and/or image) | ### Chat request body ```json { "query": "What are the side effects of this medicine?", "image_base64": "", "conversation_id": 3 } ``` --- ## Database Schema ```sql CREATE TABLE conversations ( id SERIAL PRIMARY KEY, user_id INTEGER, title VARCHAR(100), created_at TIMESTAMP DEFAULT NOW() ); CREATE TABLE messages ( id SERIAL PRIMARY KEY, conversation_id INTEGER REFERENCES conversations(id) ON DELETE CASCADE, role VARCHAR(10), -- 'user' or 'bot' text TEXT, image_path VARCHAR(255), -- URL to served image file created_at TIMESTAMP DEFAULT NOW() ); ``` --- ## Tech Stack | Layer | Technology | Why | |---|---|---| | Agent Orchestration | LangGraph | Stateful multi-agent graph with conditional routing | | LLM | Groq (Llama 3.1 8B) | Fast inference, generous free tier | | Vision | Google Gemini 2.5 Flash Lite | Multimodal β€” identifies medicines from images | | Embeddings | HuggingFace `all-MiniLM-L6-v2` | Free, runs locally, strong semantic quality | | Vector DB | ChromaDB | Lightweight, local, zero infrastructure | | Backend | FastAPI + SQLAlchemy | Async-ready, typed, auto-docs at `/docs` | | Database | PostgreSQL | Reliable relational storage with cascade deletes | | Frontend | React + Vite | Fast, component-based chat UI | | HTTP Client | Axios | Clean promise-based API calls | | Observability | LangSmith | Traces every LangGraph run end-to-end | | Config | python-dotenv | Simple `.env` file management | --- ## Design Decisions **Why conditional routing at the graph entry point?** Text and image queries need completely different first steps. A conditional edge from `START` lets LangGraph branch cleanly β€” image inputs go straight to the vision agent, text inputs go to query rewriting β€” without any `if` statements inside the agents themselves. **Why Gemini for vision instead of the main LLM?** Groq's Llama models don't support image input. Gemini Flash Lite handles vision cheaply and accurately, so the vision agent uses Gemini just for identification, then hands off a text query to the rest of the pipeline. **Why store image paths instead of base64 in the database?** Base64-encoded images are 33% larger than the raw file and bloat the database quickly. Saving images to disk and storing only the URL keeps the database lean and makes images re-loadable in restored conversations. **Why query rewriting before retrieval?** User questions are often short and vague β€” "what are its side effects?" doesn't retrieve anything useful. The rewrite agent uses conversation history to resolve pronouns and expand the query into a full, specific sentence before it hits ChromaDB. **Why optimistic title updates in the frontend?** LLM title generation adds a noticeable delay before the sidebar updates. Showing the first 5 words of the user's message instantly makes the UI feel responsive, and the real AI-generated title replaces it silently after the response arrives. --- ## Environment Variables | Variable | Required | Description | |---|---|---| | `GROQ_API_KEY` | βœ… | Groq API key for LLM inference | | `GOOGLE_API_KEY` | βœ… | Google Gemini API key for vision | | `LANGSMITH_API_KEY` | Optional | LangSmith tracing | --- ## License MIT