Download README.md from MoBkr/dosemate: direct link, hf CLI and curl.
- Browser
- Download file 14.2 kB
-
https://huggingface.co/spaces/MoBkr/dosemate/resolve/main/README.md
- Command line
-
hf download hf://spaces/MoBkr/dosemate/README.md
-
curl -L -o README.md https://huggingface.co/spaces/MoBkr/dosemate/resolve/main/README.md
title: DoseMate
emoji: π
colorFrom: green
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
π©Ί PharmaChat
A multimodal AI medical assistant that answers medicine questions β from text or images.
Type a question. Upload a photo. Get a medical answer instantly.
What Is This?
PharmaChat is a conversational medical assistant that lets you ask natural language questions about medicines β or simply photograph a medicine box and ask about it. The system identifies the medicine from the image, retrieves relevant information from a structured knowledge base, and delivers a clear, concise answer.
Built around a 4-agent LangGraph pipeline, the system routes each request through specialized agents depending on whether the input is text or an image. All conversations are persisted in PostgreSQL and served through a FastAPI backend to a clean React chat interface.
How It Works
Text Input β Rewrite Agent ββ
ββ Retriever Agent β Response Agent β Answer
Image Input β Vision Agent βββ
- Vision Agent (image path only) β uses Gemini to identify the medicine name from a photo of its packaging
- Rewrite Agent (text path only) β rewrites the user's query to be more specific and retrieval-friendly, using conversation history to resolve references like "it" or "that medicine"
- Retriever Agent β searches ChromaDB for the most relevant chunks from the medicine knowledge base
- Response Agent β reads the retrieved content and generates a concise, conversational answer
Features
π€ Multimodal Multi-Agent Pipeline
- LangGraph workflow β 4 specialized agents connected in a conditional stateful graph
- Vision input β photograph a medicine box or strip; Gemini identifies the medicine automatically
- Query rewriting β resolves follow-up references and improves retrieval accuracy before searching
- Stateful context β full conversation history passed to each agent for multi-turn awareness
π Retrieval Pipeline
- ChromaDB vector store β medicine knowledge stored as dense vectors for semantic search
- HuggingFace embeddings β
all-MiniLM-L6-v2converts text to vectors locally, no API cost - Structured JSON knowledge base β medicines stored with name, uses, dosage, side effects, and interactions
- Configurable top-k β retrieves top 3 chunks per query for focused, concise answers
πΎ Persistent Conversation History
- PostgreSQL database β every message (text and image path) saved automatically
- Auto-generated titles β conversation titles generated by the LLM from the first message
- Optimistic UI updates β title appears instantly from the first few words, replaced by the AI title after response
- Full history sidebar β all past conversations listed, click any to restore it completely
- Image persistence β uploaded images saved to disk and served as static files; displayed inline in restored chats
β‘ FastAPI Backend
- RESTful API β clean endpoints for conversations, messages, and chat
- SQLAlchemy ORM β typed database models with cascade deletes
- Static file serving β uploaded images served directly from the backend
- CORS configured β ready for local development with the React frontend
π¨ React Frontend
- Clean chat UI β user bubbles on the right, assistant on the left
- Image preview before sending β attached image shown in the input bar
- Inline image rendering β images displayed inside chat bubbles
- Right-click to delete β context menu on any conversation in the sidebar
- Typing indicator β animated dots while the assistant is generating a response
System Architecture
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β User (Browser) β
βββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββ
β
ββββββββββΌβββββββββ
β React UI β
β Vite + Axios β
β Chat bubbles β
β Sidebar historyβ
ββββββββββ¬βββββββββ
β HTTP (REST)
ββββββββββΌβββββββββ
β FastAPI β
β app.py β
β /chat β
β /conversationsβ
ββββββββββ¬βββββββββ
β
βββββββββββββΌββββββββββββ
β β
ββββββββββΌβββββββββ βββββββββββΌβββββββββ
β PostgreSQL β β LangGraph Graph β
β database.py β β workflow.py β
β Conversations β βββββββββββ¬ββββββββββ
β Messages β β
βββββββββββββββββββ ββββββββββ΄βββββββββββββββββββββββ
β β
ββββββββββΌβββββββββ βββββββββββΌβββββββββ
β Vision Agent β β Rewrite Agent β
β Gemini Flash β β Groq / Llama β
ββββββββββ¬βββββββββ βββββββββββ¬βββββββββ
β β
ββββββββββββββββ¬βββββββββββββββββ
β
ββββββββββΌβββββββββ
β Retriever Agent β
β ChromaDB β
β HuggingFace β
ββββββββββ¬βββββββββ
β
ββββββββββΌβββββββββ
β Response Agent β
β Groq / Llama β
βββββββββββββββββββ
Project Structure
.
βββ app.py # FastAPI backend β all API endpoints
βββ agents.py # Vision, rewrite, retriever, and response agents
βββ workflow.py # LangGraph pipeline (conditional 4-node graph)
βββ state.py # Shared TypedDict state
βββ prompt.py # System prompts and prompt builders
βββ database.py # SQLAlchemy models and PostgreSQL connection
βββ main.py # Entry point
βββ medicines.json # Structured medicine knowledge base
βββ notebook.ipynb # Knowledge base ingestion notebook
β
βββ frontend/ # React + Vite frontend
β βββ src/
β β βββ App.jsx # Root component β conversation state
β β βββ api.js # Axios API client
β β βββ components/
β β βββ Sidebar.jsx # Conversation list + new chat
β β βββ ChatWindow.jsx # Message bubbles + loading indicator
β β βββ InputBar.jsx # Text input + image attachment
β βββ ...
β
β
βββ .env.example # API key template
βββ requirements.txt # Python dependencies
βββ README.md
Quickstart
Prerequisites
- Python 3.10+
- Node.js 18+
- PostgreSQL running locally
- A Groq API key (free)
- A Google Gemini API key (free)
1. Clone the repository
git clone https://github.com/your-username/pharmchat.git
cd pharmchat
2. Install Python dependencies
pip install -r requirements.txt
3. Set up environment variables
cp .env.example .env
Edit .env:
GROQ_API_KEY="your_groq_key_here"
GOOGLE_API_KEY="your_gemini_key_here"
4. Set up PostgreSQL
Create a database and update the connection URL in database.py:
Database_URL = "postgresql://postgres:yourpassword@localhost:5432/medicine_assistant"
Then create the tables by running the backend once β SQLAlchemy will auto-create them on startup.
5. Ingest the knowledge base
Open notebook.ipynb and run all cells. This loads medicines.json, splits it into chunks, embeds them with HuggingFace, and stores them in ChromaDB.
Re-run this whenever you update
medicines.json.
Deletechroma_db/first to avoid duplicate entries.
6. Start the backend
uvicorn app:app --reload
Backend runs at http://localhost:8000.
7. Start the frontend
cd frontend
npm install
npm run dev
Frontend runs at http://localhost:5173.
API Reference
| Method | Endpoint | Description |
|---|---|---|
GET |
/health |
Health check |
POST |
/conversation |
Create a new conversation |
GET |
/conversations |
List all conversations |
DELETE |
/conversations/{id} |
Delete a conversation and its images |
GET |
/conversations/{id}/messages |
Load all messages for a conversation |
POST |
/chat |
Send a message (text and/or image) |
Chat request body
{
"query": "What are the side effects of this medicine?",
"image_base64": "<base64-encoded image or null>",
"conversation_id": 3
}
Database Schema
CREATE TABLE conversations (
id SERIAL PRIMARY KEY,
user_id INTEGER,
title VARCHAR(100),
created_at TIMESTAMP DEFAULT NOW()
);
CREATE TABLE messages (
id SERIAL PRIMARY KEY,
conversation_id INTEGER REFERENCES conversations(id) ON DELETE CASCADE,
role VARCHAR(10), -- 'user' or 'bot'
text TEXT,
image_path VARCHAR(255), -- URL to served image file
created_at TIMESTAMP DEFAULT NOW()
);
Tech Stack
| Layer | Technology | Why |
|---|---|---|
| Agent Orchestration | LangGraph | Stateful multi-agent graph with conditional routing |
| LLM | Groq (Llama 3.1 8B) | Fast inference, generous free tier |
| Vision | Google Gemini 2.5 Flash Lite | Multimodal β identifies medicines from images |
| Embeddings | HuggingFace all-MiniLM-L6-v2 |
Free, runs locally, strong semantic quality |
| Vector DB | ChromaDB | Lightweight, local, zero infrastructure |
| Backend | FastAPI + SQLAlchemy | Async-ready, typed, auto-docs at /docs |
| Database | PostgreSQL | Reliable relational storage with cascade deletes |
| Frontend | React + Vite | Fast, component-based chat UI |
| HTTP Client | Axios | Clean promise-based API calls |
| Observability | LangSmith | Traces every LangGraph run end-to-end |
| Config | python-dotenv | Simple .env file management |
Design Decisions
Why conditional routing at the graph entry point?
Text and image queries need completely different first steps. A conditional edge from START lets LangGraph branch cleanly β image inputs go straight to the vision agent, text inputs go to query rewriting β without any if statements inside the agents themselves.
Why Gemini for vision instead of the main LLM?
Groq's Llama models don't support image input. Gemini Flash Lite handles vision cheaply and accurately, so the vision agent uses Gemini just for identification, then hands off a text query to the rest of the pipeline.
Why store image paths instead of base64 in the database?
Base64-encoded images are 33% larger than the raw file and bloat the database quickly. Saving images to disk and storing only the URL keeps the database lean and makes images re-loadable in restored conversations.
Why query rewriting before retrieval?
User questions are often short and vague β "what are its side effects?" doesn't retrieve anything useful. The rewrite agent uses conversation history to resolve pronouns and expand the query into a full, specific sentence before it hits ChromaDB.
Why optimistic title updates in the frontend?
LLM title generation adds a noticeable delay before the sidebar updates. Showing the first 5 words of the user's message instantly makes the UI feel responsive, and the real AI-generated title replaces it silently after the response arrives.
Environment Variables
| Variable | Required | Description |
|---|---|---|
GROQ_API_KEY |
β | Groq API key for LLM inference |
GOOGLE_API_KEY |
β | Google Gemini API key for vision |
LANGSMITH_API_KEY |
Optional | LangSmith tracing |
License
MIT