HealthGPT-LoRA

HealthGPT-LoRA is a biomedical question-answering model built by fine-tuning Meta Llama 3.2 3B Instruct using QLoRA (PEFT) on the PubMedQA dataset.

This repository contains only the LoRA adapter, which can be loaded on top of the original Llama 3.2 3B Instruct model.

---

Model Details

• Base Model: Meta Llama 3.2 3B Instruct
• Fine-tuning Method: QLoRA (PEFT)
• Task: Biomedical Question Answering
• Framework: Transformers + PEFT
• Quantization: 4-bit NF4 (BitsAndBytes)
• Precision: BF16 Mixed Precision

---

Dataset

The model is trained on the PubMedQA dataset containing biomedical question-answer pairs.

Current Training Progress

• Training Samples: ~150,000
• Dataset Completion: Approximately 75%
• Training is currently in progress, with plans to continue training on the remaining dataset and further improve performance.

---

Current Evaluation Results

Metric	Score
Accuracy	93.4%
Precision	98.1%
Recall	93.4%
F1 Score	95.7%

The current model achieves approximately 12% higher accuracy than the base Llama 3.2 3B Instruct model on the evaluation dataset.

---

Planned Improvements

Upcoming milestones include:

• Retrieval-Augmented Generation (RAG)
• FAISS Vector Database
• Multi-source Medical Knowledge Retrieval
• FastAPI Backend
• Docker Deployment
• CI/CD Pipeline
• Web-based Interface

---

Usage

from transformers import AutoModelForCausalLM
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct"
)

model = PeftModel.from_pretrained(
    base_model,
    "llmithull/HealthGPT-LoRA"
)

---

Repository Contents

This repository includes:

• LoRA Adapter Weights
• Adapter Configuration
• Training Configuration

The original Llama 3.2 model is not included and must be downloaded separately from Hugging Face.

---

Project Status

🚧 Active Development

HealthGPT is an ongoing project focused on building a production-ready biomedical AI assistant. The current release represents approximately 75% of the planned fine-tuning process, with Retrieval-Augmented Generation (RAG) and deployment planned in future updates.