|
Download README.md from MoBkr/dosemate: direct link, hf CLI and curl.
- Browser
- Download file 14.2 kB
-
https://huggingface.co/spaces/MoBkr/dosemate/resolve/main/README.md
- Command line
-
hf download hf://spaces/MoBkr/dosemate/README.md
-
curl -L -o README.md https://huggingface.co/spaces/MoBkr/dosemate/resolve/main/README.md
14.2 kB
| title: DoseMate | |
| emoji: π | |
| colorFrom: green | |
| colorTo: blue | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| <div align="center"> | |
| # π©Ί PharmaChat | |
| **A multimodal AI medical assistant that answers medicine questions β from text or images.** | |
| [](https://python.org) | |
| [](https://fastapi.tiangolo.com) | |
| [](https://react.dev) | |
| [](https://langchain-ai.github.io/langgraph) | |
| [](https://trychroma.com) | |
| [](https://postgresql.org) | |
| *Type a question. Upload a photo. Get a medical answer instantly.* | |
| </div> | |
| --- | |
| ## What Is This? | |
| PharmaChat is a conversational medical assistant that lets you ask natural language questions about medicines β or simply **photograph a medicine box** and ask about it. The system identifies the medicine from the image, retrieves relevant information from a structured knowledge base, and delivers a clear, concise answer. | |
| Built around a **4-agent LangGraph pipeline**, the system routes each request through specialized agents depending on whether the input is text or an image. All conversations are persisted in **PostgreSQL** and served through a **FastAPI** backend to a clean **React** chat interface. | |
| --- | |
| ## How It Works | |
| ``` | |
| Text Input β Rewrite Agent ββ | |
| ββ Retriever Agent β Response Agent β Answer | |
| Image Input β Vision Agent βββ | |
| ``` | |
| 1. **Vision Agent** *(image path only)* β uses Gemini to identify the medicine name from a photo of its packaging | |
| 2. **Rewrite Agent** *(text path only)* β rewrites the user's query to be more specific and retrieval-friendly, using conversation history to resolve references like "it" or "that medicine" | |
| 3. **Retriever Agent** β searches ChromaDB for the most relevant chunks from the medicine knowledge base | |
| 4. **Response Agent** β reads the retrieved content and generates a concise, conversational answer | |
| --- | |
| ## Features | |
| ### π€ Multimodal Multi-Agent Pipeline | |
| - **LangGraph workflow** β 4 specialized agents connected in a conditional stateful graph | |
| - **Vision input** β photograph a medicine box or strip; Gemini identifies the medicine automatically | |
| - **Query rewriting** β resolves follow-up references and improves retrieval accuracy before searching | |
| - **Stateful context** β full conversation history passed to each agent for multi-turn awareness | |
| ### π Retrieval Pipeline | |
| - **ChromaDB vector store** β medicine knowledge stored as dense vectors for semantic search | |
| - **HuggingFace embeddings** β `all-MiniLM-L6-v2` converts text to vectors locally, no API cost | |
| - **Structured JSON knowledge base** β medicines stored with name, uses, dosage, side effects, and interactions | |
| - **Configurable top-k** β retrieves top 3 chunks per query for focused, concise answers | |
| ### πΎ Persistent Conversation History | |
| - **PostgreSQL database** β every message (text and image path) saved automatically | |
| - **Auto-generated titles** β conversation titles generated by the LLM from the first message | |
| - **Optimistic UI updates** β title appears instantly from the first few words, replaced by the AI title after response | |
| - **Full history sidebar** β all past conversations listed, click any to restore it completely | |
| - **Image persistence** β uploaded images saved to disk and served as static files; displayed inline in restored chats | |
| ### β‘ FastAPI Backend | |
| - **RESTful API** β clean endpoints for conversations, messages, and chat | |
| - **SQLAlchemy ORM** β typed database models with cascade deletes | |
| - **Static file serving** β uploaded images served directly from the backend | |
| - **CORS configured** β ready for local development with the React frontend | |
| ### π¨ React Frontend | |
| - **Clean chat UI** β user bubbles on the right, assistant on the left | |
| - **Image preview before sending** β attached image shown in the input bar | |
| - **Inline image rendering** β images displayed inside chat bubbles | |
| - **Right-click to delete** β context menu on any conversation in the sidebar | |
| - **Typing indicator** β animated dots while the assistant is generating a response | |
| --- | |
| ## System Architecture | |
| ``` | |
| βββββββββββββββββββββββββββββββββββββββββββββββββββββββ | |
| β User (Browser) β | |
| βββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββ | |
| β | |
| ββββββββββΌβββββββββ | |
| β React UI β | |
| β Vite + Axios β | |
| β Chat bubbles β | |
| β Sidebar historyβ | |
| ββββββββββ¬βββββββββ | |
| β HTTP (REST) | |
| ββββββββββΌβββββββββ | |
| β FastAPI β | |
| β app.py β | |
| β /chat β | |
| β /conversationsβ | |
| ββββββββββ¬βββββββββ | |
| β | |
| βββββββββββββΌββββββββββββ | |
| β β | |
| ββββββββββΌβββββββββ βββββββββββΌβββββββββ | |
| β PostgreSQL β β LangGraph Graph β | |
| β database.py β β workflow.py β | |
| β Conversations β βββββββββββ¬ββββββββββ | |
| β Messages β β | |
| βββββββββββββββββββ ββββββββββ΄βββββββββββββββββββββββ | |
| β β | |
| ββββββββββΌβββββββββ βββββββββββΌβββββββββ | |
| β Vision Agent β β Rewrite Agent β | |
| β Gemini Flash β β Groq / Llama β | |
| ββββββββββ¬βββββββββ βββββββββββ¬βββββββββ | |
| β β | |
| ββββββββββββββββ¬βββββββββββββββββ | |
| β | |
| ββββββββββΌβββββββββ | |
| β Retriever Agent β | |
| β ChromaDB β | |
| β HuggingFace β | |
| ββββββββββ¬βββββββββ | |
| β | |
| ββββββββββΌβββββββββ | |
| β Response Agent β | |
| β Groq / Llama β | |
| βββββββββββββββββββ | |
| ``` | |
| --- | |
| ## Project Structure | |
| ``` | |
| . | |
| βββ app.py # FastAPI backend β all API endpoints | |
| βββ agents.py # Vision, rewrite, retriever, and response agents | |
| βββ workflow.py # LangGraph pipeline (conditional 4-node graph) | |
| βββ state.py # Shared TypedDict state | |
| βββ prompt.py # System prompts and prompt builders | |
| βββ database.py # SQLAlchemy models and PostgreSQL connection | |
| βββ main.py # Entry point | |
| βββ medicines.json # Structured medicine knowledge base | |
| βββ notebook.ipynb # Knowledge base ingestion notebook | |
| β | |
| βββ frontend/ # React + Vite frontend | |
| β βββ src/ | |
| β β βββ App.jsx # Root component β conversation state | |
| β β βββ api.js # Axios API client | |
| β β βββ components/ | |
| β β βββ Sidebar.jsx # Conversation list + new chat | |
| β β βββ ChatWindow.jsx # Message bubbles + loading indicator | |
| β β βββ InputBar.jsx # Text input + image attachment | |
| β βββ ... | |
| β | |
| β | |
| βββ .env.example # API key template | |
| βββ requirements.txt # Python dependencies | |
| βββ README.md | |
| ``` | |
| --- | |
| ## Quickstart | |
| ### Prerequisites | |
| - Python 3.10+ | |
| - Node.js 18+ | |
| - PostgreSQL running locally | |
| - A [Groq API key](https://console.groq.com) (free) | |
| - A [Google Gemini API key](https://aistudio.google.com) (free) | |
| ### 1. Clone the repository | |
| ```bash | |
| git clone https://github.com/your-username/pharmchat.git | |
| cd pharmchat | |
| ``` | |
| ### 2. Install Python dependencies | |
| ```bash | |
| pip install -r requirements.txt | |
| ``` | |
| ### 3. Set up environment variables | |
| ```bash | |
| cp .env.example .env | |
| ``` | |
| Edit `.env`: | |
| ```env | |
| GROQ_API_KEY="your_groq_key_here" | |
| GOOGLE_API_KEY="your_gemini_key_here" | |
| ``` | |
| ### 4. Set up PostgreSQL | |
| Create a database and update the connection URL in `database.py`: | |
| ```python | |
| Database_URL = "postgresql://postgres:yourpassword@localhost:5432/medicine_assistant" | |
| ``` | |
| Then create the tables by running the backend once β SQLAlchemy will auto-create them on startup. | |
| ### 5. Ingest the knowledge base | |
| Open `notebook.ipynb` and run all cells. This loads `medicines.json`, splits it into chunks, embeds them with HuggingFace, and stores them in ChromaDB. | |
| > **Re-run this whenever you update `medicines.json`.** | |
| > Delete `chroma_db/` first to avoid duplicate entries. | |
| ### 6. Start the backend | |
| ```bash | |
| uvicorn app:app --reload | |
| ``` | |
| Backend runs at [http://localhost:8000](http://localhost:8000). | |
| ### 7. Start the frontend | |
| ```bash | |
| cd frontend | |
| npm install | |
| npm run dev | |
| ``` | |
| Frontend runs at [http://localhost:5173](http://localhost:5173). | |
| --- | |
| ## API Reference | |
| | Method | Endpoint | Description | | |
| |---|---|---| | |
| | `GET` | `/health` | Health check | | |
| | `POST` | `/conversation` | Create a new conversation | | |
| | `GET` | `/conversations` | List all conversations | | |
| | `DELETE` | `/conversations/{id}` | Delete a conversation and its images | | |
| | `GET` | `/conversations/{id}/messages` | Load all messages for a conversation | | |
| | `POST` | `/chat` | Send a message (text and/or image) | | |
| ### Chat request body | |
| ```json | |
| { | |
| "query": "What are the side effects of this medicine?", | |
| "image_base64": "<base64-encoded image or null>", | |
| "conversation_id": 3 | |
| } | |
| ``` | |
| --- | |
| ## Database Schema | |
| ```sql | |
| CREATE TABLE conversations ( | |
| id SERIAL PRIMARY KEY, | |
| user_id INTEGER, | |
| title VARCHAR(100), | |
| created_at TIMESTAMP DEFAULT NOW() | |
| ); | |
| CREATE TABLE messages ( | |
| id SERIAL PRIMARY KEY, | |
| conversation_id INTEGER REFERENCES conversations(id) ON DELETE CASCADE, | |
| role VARCHAR(10), -- 'user' or 'bot' | |
| text TEXT, | |
| image_path VARCHAR(255), -- URL to served image file | |
| created_at TIMESTAMP DEFAULT NOW() | |
| ); | |
| ``` | |
| --- | |
| ## Tech Stack | |
| | Layer | Technology | Why | | |
| |---|---|---| | |
| | Agent Orchestration | LangGraph | Stateful multi-agent graph with conditional routing | | |
| | LLM | Groq (Llama 3.1 8B) | Fast inference, generous free tier | | |
| | Vision | Google Gemini 2.5 Flash Lite | Multimodal β identifies medicines from images | | |
| | Embeddings | HuggingFace `all-MiniLM-L6-v2` | Free, runs locally, strong semantic quality | | |
| | Vector DB | ChromaDB | Lightweight, local, zero infrastructure | | |
| | Backend | FastAPI + SQLAlchemy | Async-ready, typed, auto-docs at `/docs` | | |
| | Database | PostgreSQL | Reliable relational storage with cascade deletes | | |
| | Frontend | React + Vite | Fast, component-based chat UI | | |
| | HTTP Client | Axios | Clean promise-based API calls | | |
| | Observability | LangSmith | Traces every LangGraph run end-to-end | | |
| | Config | python-dotenv | Simple `.env` file management | | |
| --- | |
| ## Design Decisions | |
| **Why conditional routing at the graph entry point?** | |
| Text and image queries need completely different first steps. A conditional edge from `START` lets LangGraph branch cleanly β image inputs go straight to the vision agent, text inputs go to query rewriting β without any `if` statements inside the agents themselves. | |
| **Why Gemini for vision instead of the main LLM?** | |
| Groq's Llama models don't support image input. Gemini Flash Lite handles vision cheaply and accurately, so the vision agent uses Gemini just for identification, then hands off a text query to the rest of the pipeline. | |
| **Why store image paths instead of base64 in the database?** | |
| Base64-encoded images are 33% larger than the raw file and bloat the database quickly. Saving images to disk and storing only the URL keeps the database lean and makes images re-loadable in restored conversations. | |
| **Why query rewriting before retrieval?** | |
| User questions are often short and vague β "what are its side effects?" doesn't retrieve anything useful. The rewrite agent uses conversation history to resolve pronouns and expand the query into a full, specific sentence before it hits ChromaDB. | |
| **Why optimistic title updates in the frontend?** | |
| LLM title generation adds a noticeable delay before the sidebar updates. Showing the first 5 words of the user's message instantly makes the UI feel responsive, and the real AI-generated title replaces it silently after the response arrives. | |
| --- | |
| ## Environment Variables | |
| | Variable | Required | Description | | |
| |---|---|---| | |
| | `GROQ_API_KEY` | β | Groq API key for LLM inference | | |
| | `GOOGLE_API_KEY` | β | Google Gemini API key for vision | | |
| | `LANGSMITH_API_KEY` | Optional | LangSmith tracing | | |
| --- | |
| ## License | |
| MIT | |