This Python script demonstrates a minimal setup for using the ChatCM-RAG (Retrieval-Augmented Generation) system to search biomedical literature using natural language queries. It combines semantic search via a sentence-transformer model with a pre-built FAISS index and structured metadata to return the most relevant research abstracts from a curated dataset.
The code first downloads the full ChatMed-RAG package from Hugging Face using snapshot_download(repo_id="fc28/ChatMed-RAG"). It then initializes three components: a sentence encoder (all-mpnet-base-v2) to embed queries, a FAISS index (index.faiss) to search for semantically similar documents, and a metadata file (medllm_metadata.csv) that contains information like titles, abstracts, PMIDs, and publication years.
The search() function accepts a query and returns the top-k most relevant results by performing vector similarity search. Each result includes the title, year, PMID, and a truncated abstract snippet.
# Medical Literature RAG System
A comprehensive Retrieval-Augmented Generation (RAG) system specifically designed for medical literature analysis and question-answering, with advanced topic modeling and evaluation capabilities.
## Overview
This project implements a state-of-the-art RAG pipeline that processes medical literature from PubMed data, performs intelligent topic clustering using BERTopic, and provides accurate question-answering capabilities for medical research queries. The system is particularly optimized for ChatGPT and AI applications in healthcare research.
## Key Features
### **Advanced Data Processing**
- **Multi-format Support**: Processes Excel files containing medical literature metadata
- **Smart Data Cleaning**: Automatic deduplication, standardization, and validation
- **Structured Record Management**: Handles PMID, abstracts, MeSH terms, keywords, and citations
### **Intelligent Topic Modeling**
- **BERTopic Integration**: Advanced topic discovery usi…
