| --- |
| library_name: peft |
| base_model: microsoft/phi-2 |
| tags: |
| - biomedical |
| - summarization |
| - lay-summary |
| - healthcare |
| - nlp |
| - fine-tuned |
| - lora |
| - peft |
| - elife |
| - plos |
| - medical-text |
| language: |
| - en |
| license: mit |
| metrics: |
| - rouge |
| - bertscore |
| - readability |
| datasets: |
| - elife |
| - plos |
| pipeline_tag: text2text-generation |
| --- |
| |
| # Phi-2 BioLaySum: Biomedical Lay Summarization Model 🏆 |
|
|
| ## 📖 Model Overview |
|
|
| **Phi-2 BioLaySum** is a **champion model** that emerged as the most efficient and highest-performing solution for generating lay summaries of biomedical articles. This model converts complex medical research into easily understandable summaries for the general public, significantly enhancing accessibility to scientific literature. |
|
|
| **🥇 Key Achievement**: This model **outperformed** T5-Base, T5-Large, FlanT5-Base, BioGPT, and Falconsi-Medical_summarisation across all evaluation dimensions (relevance, readability, and factuality) while maintaining optimal computational efficiency. |
| |
| ## 🎯 Model Purpose |
| |
| This model addresses the critical need to bridge the gap between complex biomedical research and public health literacy by: |
| - Converting medical articles into patient-friendly summaries |
| - Supporting healthcare communication between professionals and patients |
| - Enhancing public access to biomedical research findings |
| - Enabling better-informed health decisions by the general public |
| |
| ## 🏗️ Model Architecture |
| |
| - **Base Model**: microsoft/phi-2 |
| - **Fine-tuning Technique**: LoRA (Low-Rank Adaptation) + PEFT (Parameter Efficient Fine-tuning) |
| - **Model Type**: Text-to-Text Generation (Summarization) |
| - **Language**: English |
| - **Domain**: Biomedical/Healthcare |
| |
| ## 📊 Performance Highlights |
| |
| ### Why Phi-2 is the Champion Model: |
| - ✅ **Superior Performance**: Best scores across relevance, readability, and factuality metrics |
| - ✅ **Resource Efficiency**: Optimal performance-to-resource ratio |
| - ✅ **Compact Size**: Most efficient in terms of model size and computational requirements |
| - ✅ **Cost-Effective**: Best balance of quality and computational cost |
| |
| ### Evaluation Results: |
| - **Relevance**: Measured using ROUGE (1, 2, L) and BERTScore |
| - **Readability**: Assessed via Flesch-Kincaid Grade Level (FKGL) and Dale-Chall Readability Score (DCRS) |
| - **Factuality**: Verified using BARTScore and factual consistency checks |
| |
| ## 🚀 Quick Start |
| |
| ### Loading the Model |
| |
| ```python |
| from peft import PeftModel, PeftConfig |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| import torch |
| |
| # Load the base model and tokenizer |
| base_model_name = "microsoft/phi-2" |
| model = AutoModelForCausalLM.from_pretrained( |
| base_model_name, |
| torch_dtype=torch.float16, |
| device_map="auto" |
| ) |
| tokenizer = AutoTokenizer.from_pretrained(base_model_name) |
| |
| # Load the fine-tuned adapter |
| model = PeftModel.from_pretrained(model, "sank29mane/phi-2-biolaysum") |
| |
| # Set padding token |
| if tokenizer.pad_token is None: |
| tokenizer.pad_token = tokenizer.eos_token |
| ``` |
| |
| ### Generating Lay Summaries |
|
|
| ```python |
| def generate_lay_summary(medical_text, max_length=150): |
| # Prepare input |
| prompt = f"Summarize the following medical text for a general audience: {medical_text}" |
| inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=512) |
| |
| # Generate summary |
| with torch.no_grad(): |
| outputs = model.generate( |
| **inputs, |
| max_length=max_length, |
| temperature=0.7, |
| do_sample=True, |
| pad_token_id=tokenizer.eos_token_id |
| ) |
| |
| # Decode and return |
| summary = tokenizer.decode(outputs[0], skip_special_tokens=True) |
| return summary.split(":")[-1].strip() # Extract generated part |
| |
| # Example usage |
| medical_text = """ |
| The study investigated the efficacy of novel therapeutic interventions |
| in cardiovascular disease management through randomized controlled trials... |
| """ |
| |
| lay_summary = generate_lay_summary(medical_text) |
| print(f"Lay Summary: {lay_summary}") |
| ``` |
|
|
| ## 📚 Training Details |
|
|
| ### Training Data |
| - **eLife Dataset**: Open-access biomedical research articles with lay summaries |
| - **PLOS Dataset**: Public Library of Science biomedical publications |
| - **Data Processing**: Advanced preprocessing for optimal model performance |
|
|
| ### Training Configuration |
| - **Fine-tuning Method**: LoRA (Low-Rank Adaptation) with PEFT |
| - **Base Model**: microsoft/phi-2 |
| - **Training Framework**: PyTorch + Hugging Face Transformers |
| - **Optimization**: Parameter-efficient approach reducing computational requirements |
|
|
| ### Training Advantages |
| - **Efficiency**: LoRA reduces trainable parameters while maintaining performance |
| - **Resource-Friendly**: PEFT enables high-quality fine-tuning with limited resources |
| - **Stability**: Advanced techniques ensure robust model behavior |
|
|
| ## 📈 Comparative Analysis |
|
|
| ### Models Compared: |
| 1. **T5-Base** - Text-to-Text Transfer Transformer (Base) |
| 2. **T5-Large** - Text-to-Text Transfer Transformer (Large) |
| 3. **FlanT5-Base** - Instruction-tuned T5 model |
| 4. **BioGPT** - Biomedical domain-specific GPT |
| 5. **Phi-2** - Microsoft's efficient language model (**Winner**) |
| 6. **Falconsi-Medical_summarisation** - Specialized medical summarization model |
| |
| ### Key Findings: |
| - **Phi-2 outperformed all competitors** in comprehensive evaluation |
| - **Domain-specific models** (BioGPT, Falconsi) showed advantages over general T5 models |
| - **Parameter efficiency** of Phi-2 provided superior cost-effectiveness |
| - **Smaller models** can achieve better performance with proper fine-tuning |
| |
| ## 🎯 Use Cases |
| |
| ### Healthcare Applications: |
| - **Patient Education**: Convert research findings into understandable format |
| - **Medical Communication**: Support doctor-patient conversations |
| - **Health Journalism**: Assist science writers and health reporters |
| - **Educational Materials**: Create teaching resources for health education |
| - **Policy Support**: Provide accessible summaries for health policy decisions |
| |
| ### Target Audiences: |
| - Healthcare professionals seeking patient communication tools |
| - Patients and families researching medical conditions |
| - Health educators and trainers |
| - Medical journalists and science communicators |
| - Public health policy makers |
| |
| ## ⚡ Performance Metrics |
| |
| ### Evaluation Framework: |
| - **ROUGE Scores**: Overlap-based relevance assessment |
| - **BERTScore**: Semantic similarity evaluation |
| - **Readability Metrics**: FKGL and DCRS for accessibility |
| - **Factual Consistency**: BARTScore for accuracy verification |
| |
| ### Resource Efficiency: |
| - **Model Size**: Compact and deployment-friendly |
| - **Inference Speed**: Fast generation suitable for real-time applications |
| - **Memory Usage**: Optimized for various computational environments |
| - **Cost Effectiveness**: Best performance per computational dollar |
| |
| ## 🔧 Technical Specifications |
| |
| ### Model Details: |
| - **Architecture**: Transformer-based with LoRA adaptation |
| - **Parameters**: Base Phi-2 + efficient LoRA adapters |
| - **Precision**: Mixed precision training for efficiency |
| - **Framework**: PyTorch with Hugging Face ecosystem |
| |
| ### System Requirements: |
| - **Minimum GPU**: 4GB VRAM for inference |
| - **Recommended**: 8GB+ VRAM for optimal performance |
| - **CPU**: Compatible with CPU inference (slower) |
| - **Dependencies**: transformers, peft, torch |
| |
| ## 📖 Research Impact |
| |
| This model contributes to: |
| - **Democratizing Medical Knowledge**: Making research accessible to all |
| - **Advancing Healthcare NLP**: Pushing boundaries of medical text processing |
| - **Resource-Efficient AI**: Demonstrating effective use of LoRA and PEFT |
| - **Evaluation Methodology**: Comprehensive framework for summarization assessment |
| |
| ## 📄 License & Citation |
| |
| ### License |
| This model is released under the **MIT License**, promoting open research and development. |
| |
| ### Citation |
| If you use this model in your research, please cite: |
| |
| ```bibtex |
| @misc{mane2024phi2biolaysum, |
| title={Phi-2 BioLaySum: Resource-Efficient Biomedical Lay Summarization using LoRA and PEFT}, |
| author={Mane, Sanket}, |
| year={2024}, |
| publisher={Hugging Face}, |
| url={https://huggingface.co/sank29mane/phi-2-biolaysum} |
| } |
| ``` |
| |
| ## 🔗 Related Resources |
| |
| - **GitHub Repository**: [lays-bio-summery](https://github.com/sank29mane/lays-bio-summery) - Complete training code and evaluation |
| - **Base Model**: [microsoft/phi-2](https://huggingface.co/microsoft/phi-2) |
| - **Research Paper**: [Detailed methodology and results](https://github.com/sank29mane/lays-bio-summery) |
| |
| ## 👨💻 Author |
| |
| **Sanket Mane** - [@sank29mane](https://github.com/sank29mane) |
| *Researcher in Biomedical NLP and Efficient Language Models* |
| |
| ## 📞 Contact & Support |
| |
| - **GitHub Issues**: [Create an issue](https://github.com/sank29mane/lays-bio-summery/issues) |
| - **Model Issues**: Use the Community tab above |
| - **Research Collaborations**: Through GitHub profile |
| |
| ## 🚨 Limitations & Considerations |
| |
| ### Current Limitations: |
| - **Language**: Currently optimized for English biomedical text |
| - **Domain**: Focused on general biomedical research (not clinical notes) |
| - **Length**: Optimized for article-length inputs, may vary with very short/long texts |
| |
| ### Recommended Use: |
| - Use for biomedical research article summarization |
| - Validate outputs for critical healthcare decisions |
| - Consider human review for patient-facing applications |
| |
| ## 🔄 Model Updates |
| |
| - **v1.0**: Initial release with LoRA+PEFT fine-tuning |
| - **Future**: Planned improvements for multi-language support and clinical text adaptation |
| |
| --- |
| |
| ### Framework Versions |
| - **PEFT**: 0.7.2.dev0 |
| - **Transformers**: Compatible with latest versions |
| - **PyTorch**: 1.12+ |
| |
| ⭐ **Star this model if you find it useful for your biomedical NLP research!** ⭐ |
| |
