| --- |
| license: gemma |
| base_model: google/gemma-3-27b-it |
| library_name: peft |
| pipeline_tag: text-generation |
| tags: |
| - lora |
| - peft |
| - sft |
| - medical |
| - clinical-nlp |
| - citation |
| - refqa |
| --- |
| |
| # RefQA — Gemma-3-27B mixed adapter (v0.7) |
|
|
| A LoRA adapter that fine-tunes **google/gemma-3-27b-it** on a mix of medical instruction data and the [RefQA](https://doi.org/10.5281/zenodo.20805692) CitationQA data. It learns the RefQA CitationQA task while retaining general medical question-answering ability. |
|
|
| ## Task |
|
|
| Two capabilities from one adapter: |
| 1. Structured **CitationQA** generation on the RefQA schema (given the RefQA system prompt and a citation context), and |
| 2. Retained general medical question answering. |
|
|
| ## Results |
|
|
| - Final `eval_loss` 0.215, token accuracy 92.6% (2 epochs / 50,224 steps) |
| - Out-of-domain evaluation: **PubMedQA 74.10%**, **MedQA 24.59%** |
|
|
| These numbers are reported as measured, not as a claim of state of the art. |
|
|
| ## Limitations |
|
|
| - Research artifact. No clinical validation; not for clinical decision-making. |
| - The CitationQA behavior requires the RefQA system prompt and chat format (below); without them the structured output is not reproduced. |
|
|
| ## Training |
|
|
| - Base: `google/gemma-3-27b-it` (snapshot `005ad340`) |
| - Data: medical instruction mix + RefQA |
| - LoRA: r=64, alpha=128, dropout=0.05; language-model attention and MLP projections |
| - 2 epochs / 50,224 steps |
| - Stack: torch 2.11 (cu130), transformers 5.7, peft 0.19, trl 1.3 |
|
|
| ## How to use |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| from peft import PeftModel |
| |
| adapter = "taejoon89/refqa" |
| base = "google/gemma-3-27b-it" # gated: accept the Gemma license first |
| |
| tok = AutoTokenizer.from_pretrained(adapter) |
| model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto") |
| model = PeftModel.from_pretrained(model, adapter) |
| ``` |
|
|
| For CitationQA generation, the system prompt and input builder are in `extract_qa_glm_v03.py` at https://github.com/jin-0311/refqa. The training chat template is included here as `chat_template.jinja`. |
|
|
| ## Dataset and code |
|
|
| - Dataset: RefQA — https://doi.org/10.5281/zenodo.20805692 |
| - Pipeline code: https://github.com/jin-0311/refqa |
|
|
| ## License |
|
|
| Use is governed by the base model's license (Gemma Terms of Use). You must accept the Gemma license to download and use the base model. |
|
|
