dosemate / README.md
DoseMate
Fix HF Space colorTo metadata
4d0c7fc
|
Raw History Blame Contribute Delete
14.2 kB
metadata
title: DoseMate
emoji: πŸ’Š
colorFrom: green
colorTo: blue
sdk: docker
app_port: 7860
pinned: false

🩺 PharmaChat

A multimodal AI medical assistant that answers medicine questions β€” from text or images.

Python FastAPI React LangGraph ChromaDB PostgreSQL

Type a question. Upload a photo. Get a medical answer instantly.


What Is This?

PharmaChat is a conversational medical assistant that lets you ask natural language questions about medicines β€” or simply photograph a medicine box and ask about it. The system identifies the medicine from the image, retrieves relevant information from a structured knowledge base, and delivers a clear, concise answer.

Built around a 4-agent LangGraph pipeline, the system routes each request through specialized agents depending on whether the input is text or an image. All conversations are persisted in PostgreSQL and served through a FastAPI backend to a clean React chat interface.


How It Works

Text Input  β†’ Rewrite Agent  ─┐
                               β”œβ†’ Retriever Agent β†’ Response Agent β†’ Answer
Image Input β†’ Vision Agent  β”€β”€β”˜
  1. Vision Agent (image path only) β€” uses Gemini to identify the medicine name from a photo of its packaging
  2. Rewrite Agent (text path only) β€” rewrites the user's query to be more specific and retrieval-friendly, using conversation history to resolve references like "it" or "that medicine"
  3. Retriever Agent β€” searches ChromaDB for the most relevant chunks from the medicine knowledge base
  4. Response Agent β€” reads the retrieved content and generates a concise, conversational answer

Features

πŸ€– Multimodal Multi-Agent Pipeline

  • LangGraph workflow β€” 4 specialized agents connected in a conditional stateful graph
  • Vision input β€” photograph a medicine box or strip; Gemini identifies the medicine automatically
  • Query rewriting β€” resolves follow-up references and improves retrieval accuracy before searching
  • Stateful context β€” full conversation history passed to each agent for multi-turn awareness

πŸ” Retrieval Pipeline

  • ChromaDB vector store β€” medicine knowledge stored as dense vectors for semantic search
  • HuggingFace embeddings β€” all-MiniLM-L6-v2 converts text to vectors locally, no API cost
  • Structured JSON knowledge base β€” medicines stored with name, uses, dosage, side effects, and interactions
  • Configurable top-k β€” retrieves top 3 chunks per query for focused, concise answers

πŸ’Ύ Persistent Conversation History

  • PostgreSQL database β€” every message (text and image path) saved automatically
  • Auto-generated titles β€” conversation titles generated by the LLM from the first message
  • Optimistic UI updates β€” title appears instantly from the first few words, replaced by the AI title after response
  • Full history sidebar β€” all past conversations listed, click any to restore it completely
  • Image persistence β€” uploaded images saved to disk and served as static files; displayed inline in restored chats

⚑ FastAPI Backend

  • RESTful API β€” clean endpoints for conversations, messages, and chat
  • SQLAlchemy ORM β€” typed database models with cascade deletes
  • Static file serving β€” uploaded images served directly from the backend
  • CORS configured β€” ready for local development with the React frontend

🎨 React Frontend

  • Clean chat UI β€” user bubbles on the right, assistant on the left
  • Image preview before sending β€” attached image shown in the input bar
  • Inline image rendering β€” images displayed inside chat bubbles
  • Right-click to delete β€” context menu on any conversation in the sidebar
  • Typing indicator β€” animated dots while the assistant is generating a response

System Architecture

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                   User (Browser)                    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                      β”‚
             β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
             β”‚    React UI     β”‚
             β”‚  Vite + Axios   β”‚
             β”‚  Chat bubbles   β”‚
             β”‚  Sidebar historyβ”‚
             β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                      β”‚  HTTP (REST)
             β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
             β”‚   FastAPI       β”‚
             β”‚   app.py        β”‚
             β”‚   /chat         β”‚
             β”‚   /conversationsβ”‚
             β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                      β”‚
          β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
          β”‚                       β”‚
 β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
 β”‚   PostgreSQL    β”‚   β”‚  LangGraph Graph  β”‚
 β”‚   database.py   β”‚   β”‚  workflow.py      β”‚
 β”‚   Conversations β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
 β”‚   Messages      β”‚             β”‚
 β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                        β”‚                               β”‚
               β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”           β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
               β”‚  Vision Agent   β”‚           β”‚  Rewrite Agent   β”‚
               β”‚  Gemini Flash   β”‚           β”‚  Groq / Llama    β”‚
               β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜           β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                        β”‚                               β”‚
                        β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                       β”‚
                              β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
                              β”‚ Retriever Agent β”‚
                              β”‚   ChromaDB      β”‚
                              β”‚   HuggingFace   β”‚
                              β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                       β”‚
                              β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
                              β”‚ Response Agent  β”‚
                              β”‚  Groq / Llama   β”‚
                              β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Project Structure

.
β”œβ”€β”€ app.py              # FastAPI backend β€” all API endpoints
β”œβ”€β”€ agents.py           # Vision, rewrite, retriever, and response agents
β”œβ”€β”€ workflow.py         # LangGraph pipeline (conditional 4-node graph)
β”œβ”€β”€ state.py            # Shared TypedDict state
β”œβ”€β”€ prompt.py           # System prompts and prompt builders
β”œβ”€β”€ database.py         # SQLAlchemy models and PostgreSQL connection
β”œβ”€β”€ main.py             # Entry point
β”œβ”€β”€ medicines.json      # Structured medicine knowledge base
β”œβ”€β”€ notebook.ipynb      # Knowledge base ingestion notebook
β”‚
β”œβ”€β”€ frontend/           # React + Vite frontend
β”‚   β”œβ”€β”€ src/
β”‚   β”‚   β”œβ”€β”€ App.jsx         # Root component β€” conversation state
β”‚   β”‚   β”œβ”€β”€ api.js          # Axios API client
β”‚   β”‚   └── components/
β”‚   β”‚       β”œβ”€β”€ Sidebar.jsx     # Conversation list + new chat
β”‚   β”‚       β”œβ”€β”€ ChatWindow.jsx  # Message bubbles + loading indicator
β”‚   β”‚       └── InputBar.jsx    # Text input + image attachment
β”‚   └── ...
β”‚
β”‚
β”œβ”€β”€ .env.example        # API key template
β”œβ”€β”€ requirements.txt    # Python dependencies
└── README.md

Quickstart

Prerequisites

1. Clone the repository

git clone https://github.com/your-username/pharmchat.git
cd pharmchat

2. Install Python dependencies

pip install -r requirements.txt

3. Set up environment variables

cp .env.example .env

Edit .env:

GROQ_API_KEY="your_groq_key_here"
GOOGLE_API_KEY="your_gemini_key_here"

4. Set up PostgreSQL

Create a database and update the connection URL in database.py:

Database_URL = "postgresql://postgres:yourpassword@localhost:5432/medicine_assistant"

Then create the tables by running the backend once β€” SQLAlchemy will auto-create them on startup.

5. Ingest the knowledge base

Open notebook.ipynb and run all cells. This loads medicines.json, splits it into chunks, embeds them with HuggingFace, and stores them in ChromaDB.

Re-run this whenever you update medicines.json.
Delete chroma_db/ first to avoid duplicate entries.

6. Start the backend

uvicorn app:app --reload

Backend runs at http://localhost:8000.

7. Start the frontend

cd frontend
npm install
npm run dev

Frontend runs at http://localhost:5173.


API Reference

Method Endpoint Description
GET /health Health check
POST /conversation Create a new conversation
GET /conversations List all conversations
DELETE /conversations/{id} Delete a conversation and its images
GET /conversations/{id}/messages Load all messages for a conversation
POST /chat Send a message (text and/or image)

Chat request body

{
  "query": "What are the side effects of this medicine?",
  "image_base64": "<base64-encoded image or null>",
  "conversation_id": 3
}

Database Schema

CREATE TABLE conversations (
    id         SERIAL PRIMARY KEY,
    user_id    INTEGER,
    title      VARCHAR(100),
    created_at TIMESTAMP DEFAULT NOW()
);

CREATE TABLE messages (
    id              SERIAL PRIMARY KEY,
    conversation_id INTEGER REFERENCES conversations(id) ON DELETE CASCADE,
    role            VARCHAR(10),       -- 'user' or 'bot'
    text            TEXT,
    image_path      VARCHAR(255),      -- URL to served image file
    created_at      TIMESTAMP DEFAULT NOW()
);

Tech Stack

Layer Technology Why
Agent Orchestration LangGraph Stateful multi-agent graph with conditional routing
LLM Groq (Llama 3.1 8B) Fast inference, generous free tier
Vision Google Gemini 2.5 Flash Lite Multimodal β€” identifies medicines from images
Embeddings HuggingFace all-MiniLM-L6-v2 Free, runs locally, strong semantic quality
Vector DB ChromaDB Lightweight, local, zero infrastructure
Backend FastAPI + SQLAlchemy Async-ready, typed, auto-docs at /docs
Database PostgreSQL Reliable relational storage with cascade deletes
Frontend React + Vite Fast, component-based chat UI
HTTP Client Axios Clean promise-based API calls
Observability LangSmith Traces every LangGraph run end-to-end
Config python-dotenv Simple .env file management

Design Decisions

Why conditional routing at the graph entry point?
Text and image queries need completely different first steps. A conditional edge from START lets LangGraph branch cleanly β€” image inputs go straight to the vision agent, text inputs go to query rewriting β€” without any if statements inside the agents themselves.

Why Gemini for vision instead of the main LLM?
Groq's Llama models don't support image input. Gemini Flash Lite handles vision cheaply and accurately, so the vision agent uses Gemini just for identification, then hands off a text query to the rest of the pipeline.

Why store image paths instead of base64 in the database?
Base64-encoded images are 33% larger than the raw file and bloat the database quickly. Saving images to disk and storing only the URL keeps the database lean and makes images re-loadable in restored conversations.

Why query rewriting before retrieval?
User questions are often short and vague β€” "what are its side effects?" doesn't retrieve anything useful. The rewrite agent uses conversation history to resolve pronouns and expand the query into a full, specific sentence before it hits ChromaDB.

Why optimistic title updates in the frontend?
LLM title generation adds a noticeable delay before the sidebar updates. Showing the first 5 words of the user's message instantly makes the UI feel responsive, and the real AI-generated title replaces it silently after the response arrives.


Environment Variables

Variable Required Description
GROQ_API_KEY βœ… Groq API key for LLM inference
GOOGLE_API_KEY βœ… Google Gemini API key for vision
LANGSMITH_API_KEY Optional LangSmith tracing

License

MIT