File size: 2,340 Bytes
cdac5f8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | ---
license: gemma
base_model: google/gemma-3-27b-it
library_name: peft
pipeline_tag: text-generation
tags:
- lora
- peft
- sft
- medical
- clinical-nlp
- citation
- refqa
---
# RefQA — Gemma-3-27B mixed adapter (v0.7)
A LoRA adapter that fine-tunes **google/gemma-3-27b-it** on a mix of medical instruction data and the [RefQA](https://doi.org/10.5281/zenodo.20805692) CitationQA data. It learns the RefQA CitationQA task while retaining general medical question-answering ability.
## Task
Two capabilities from one adapter:
1. Structured **CitationQA** generation on the RefQA schema (given the RefQA system prompt and a citation context), and
2. Retained general medical question answering.
## Results
- Final `eval_loss` 0.215, token accuracy 92.6% (2 epochs / 50,224 steps)
- Out-of-domain evaluation: **PubMedQA 74.10%**, **MedQA 24.59%**
These numbers are reported as measured, not as a claim of state of the art.
## Limitations
- Research artifact. No clinical validation; not for clinical decision-making.
- The CitationQA behavior requires the RefQA system prompt and chat format (below); without them the structured output is not reproduced.
## Training
- Base: `google/gemma-3-27b-it` (snapshot `005ad340`)
- Data: medical instruction mix + RefQA
- LoRA: r=64, alpha=128, dropout=0.05; language-model attention and MLP projections
- 2 epochs / 50,224 steps
- Stack: torch 2.11 (cu130), transformers 5.7, peft 0.19, trl 1.3
## How to use
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
adapter = "taejoon89/refqa"
base = "google/gemma-3-27b-it" # gated: accept the Gemma license first
tok = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter)
```
For CitationQA generation, the system prompt and input builder are in `extract_qa_glm_v03.py` at https://github.com/jin-0311/refqa. The training chat template is included here as `chat_template.jinja`.
## Dataset and code
- Dataset: RefQA — https://doi.org/10.5281/zenodo.20805692
- Pipeline code: https://github.com/jin-0311/refqa
## License
Use is governed by the base model's license (Gemma Terms of Use). You must accept the Gemma license to download and use the base model.
|